Create species, individuals, tags, sensors and link them all
tagged_individuals.RmdInitial Setup and Api Credentials
Please see station_and_receptors
for further details about how to make available the showcased package
temploapiclient to this vignette.
project_folder <- dirname(getwd())
path_to_env <- file.path(project_folder, ".env")
readRenviron(path_to_env)
api_base_url <- Sys.getenv("API_BASE_URL")
api_headers <- list("Accept" = "application/json", "Content-type" = "application/json")
api_auth_token <- Sys.getenv("AUTH_TOKEN")
if (!length(api_auth_token)) {
api_auth_token <- NULL
}
templo_client <- temploapiclient::TemploApiClient$new(api_base_url, api_headers, api_auth_token)
library("magrittr") # Make '%>%' available throughout the vignetteInitial Setup and Api Credentials
In order to add tagged individuals to the database and given that we are using a relational databases, the four elements making up the relationship should be inserted one by one and in chronological order as follows:
- Insert first species if it is not yet in the database
- Insert associated sensors with tags
- Insert tags
- Insert individuals
- Establish the necessary links between them all.
Fortunately for us, the API has been shipped with a method that wraps up the last 4 actions allowing them to be completed at once.
Creating species, individuals, tags, sensors and linking them all
Species table contains two unique names: scientific_name
and common_name. This means that such names cannot be
duplicated in any way across, for example, two records.
ret <- templo_client$add_single_record_and_fetch_id(end_point = "species/", data = list(
scientific_name = "Trisopterus luscus",
common_name = "Faneca"
))
retThen we double-check that faneca has been sent and stored successfully.
api_data <- templo_client$get_dataframe_end_point(end_point = "species/")
#> INFO [2024-11-14 17:08:21] Fetching data from api... This may take a few seconds
#> INFO [2024-11-14 17:08:21] --->Finished
api_data %>%
dplyr::filter(common_name == "Faneca") %>%
assertr::verify(nrow(.) == 1) # It will halt the execution and print an error if such record does not exist
#> scientific_name common_name id fao
#> 1 Trisopterus luscus Faneca 13 NACreate sensor type
Each tag is associated to certain type of sensors that measure oceanographic variables such as pressure or temperature. As for species, we need to ensure that such sensor types already exist in the database before trying associate a sensor of that kind to a tag and in turn, this to an individual.
As the code below shows, there are already different types of sensors such as PRESSURE, TEMPERATURE or DEPTH.
api_data <- templo_client$get_dataframe_end_point(end_point = "sensortypes/")
#> INFO [2024-11-14 17:08:21] Fetching data from api... This may take a few seconds
#> INFO [2024-11-14 17:08:21] --->Finished
api_data
#> id type_name
#> 1 4 ACTIVITY
#> 2 5 CONDUCTIVITY
#> 3 1 DEPTH
#> 4 6 MORTALITY
#> 5 17 NUEVO
#> 6 38 PRESSURE
#> 7 39 SALINITY
#> 8 7 string
#> 9 3 TEMPERATURE
#> 10 2 TILTLet’s assume that we wanted to add another one to measure salinity. As for species this would be as:
ret <- templo_client$add_single_record_and_fetch_id(end_point = "sensortypes/", data = list(
type_name = "SALINITY"
))
retWe check up that the record has been sent successfully.
api_data <- templo_client$get_dataframe_end_point(end_point = "sensortypes/")
#> INFO [2024-11-14 17:08:22] Fetching data from api... This may take a few seconds
#> INFO [2024-11-14 17:08:22] --->Finished
api_data %>%
dplyr::filter(type_name == "SALINITY") %>%
assertr::verify(nrow(.) == 1)
#> id type_name
#> 1 39 SALINITYPlease notice that the field type_name is unique so that
there cannot be duplicated measured variables.
‘tagged_individuals’ dataframe build-up
From the previous insertion we identified that the species faneca was
added with id=13 and that sensor ids are
SALINITY = 39,
PRESSURE = 38,TEMPERATURE = 3. Next, we will
create a dataframe that contains the data of the individuals we wish to
insert, along with their associated tagging details.
Let’s assume a scenario where we want to add two individuals, each
belonging to a different species Trisopterus lucus and
Raja undulata. The summary of such scenario is as
follows:
- Trisopterus luscus
- Sensors DEPTH and PRESSURE
- Tag is external and active
- Raja undulata
- Sensors SALINITY
- Tag is internal and active
In the next steps, we will create individual datasets for each species and combine them into a single tagged_individuals dataframe to be sent to the API. Notice that this step-by-step guide is not required as you can create the whole dataframe at once by means you consider more suitable.
Built the sensor dataframe
We need to create a two-column dataframe that includes each sensor’s
id, as retrieved from THELMA, along with its type
(sensor). Next, we’ll construct a dataframe with three rows
to represent the sensors: two for faneca (which has two sensors) and one
for ray.
datasets <- list()
sensor_prefix <- "sensor_" # prefix that indicates the logic that fields belong to sensor category
sensor_ids <- sample(11700:11800, 3) # 3 random IDS with the hope none of them are in the db already
datasets[[length(datasets) + 1]] <- data.frame(merged_id = 1:3) %>% # to merge with following datasets.
dplyr::mutate(
!!paste0(sensor_prefix, "id") := sensor_ids,
!!paste0(sensor_prefix, "sensor") := c("DEPTH", "PRESSURE", "SALINITY"),
)
datasets[[1]]
#> merged_id sensor_id sensor_sensor
#> 1 1 11744 DEPTH
#> 2 2 11722 PRESSURE
#> 3 3 11775 SALINITYBuilt the tag dataframe on top of sensor’s
As for sensors we need to use a prefix to build the tag columns which
we will name tag_. The fields required for this dataset are
serial_number, frequency and
comm_protocol. All three fields are expected to be strings
and are obtained from THELMA. Notice that all individuals whose tags
have multiple sensors do need to have the same
serial_number. The dataframe is constructed as follows:
tag_prefix <- "tag_"
# Again, we build two random alphanumeric ids for shwocasing purposes
random_strings <- stringi::stri_rand_strings(2, 4)
serial_ids <- paste0("vignette_", c(rep(random_strings[1], 2), random_strings[2]))
frequency <- rep("69 kHz", 3)
comm_protocol <- rep("Ops", 3)
datasets[[length(datasets) + 1]] <- data.frame(merged_id = 1:3) %>%
dplyr::mutate(
!!paste0(tag_prefix, "serial_number") := serial_ids,
!!paste0(tag_prefix, "frequency") := frequency,
!!paste0(tag_prefix, "comm_protocol") := comm_protocol
)
datasets[[2]]
#> merged_id tag_serial_number tag_frequency tag_comm_protocol
#> 1 1 vignette_90SU 69 kHz Ops
#> 2 2 vignette_90SU 69 kHz Ops
#> 3 3 vignette_Hjls 69 kHz OpsBuilt the individual dataframe
Likewise we need to build the data for each individual which fields
are as follows: 1. individual_id 2.
scientific_name 3. common_name 4.
sex 5. total_length
options(width = 600)
individual_prefix <- "ind_"
# Again, we build three random alphanumeric ids for shwocasing purposes
individual_ids <- paste0("MOVE-RUN-", c(rep("fane-01", 2), "raj-01"))
scientific_names <- c(rep("Trisopterus luscus", 2), "Raja undulata")
common_names <- c(rep("Faneca", 2), "Undulate ray")
sex <- c(1, 1, 2) # two males and one females. We don't know yet what should be assigned for male and female
total_lengths <- c(35, 35, 80) # in cms (first two refer to the same individual)
datasets[[length(datasets) + 1]] <- data.frame(merged_id = 1:3) %>%
dplyr::mutate(
!!paste0(individual_prefix, "individual_id") := individual_ids,
!!paste0(individual_prefix, "scientific_name") := scientific_names,
!!paste0(individual_prefix, "common_name") := common_names,
!!paste0(individual_prefix, "sex") := sex,
!!paste0(individual_prefix, "total_length") := total_lengths,
)
datasets[[3]]
#> merged_id ind_individual_id ind_scientific_name ind_common_name ind_sex ind_total_length
#> 1 1 MOVE-RUN-fane-01 Trisopterus luscus Faneca 1 35
#> 2 2 MOVE-RUN-fane-01 Trisopterus luscus Faneca 1 35
#> 3 3 MOVE-RUN-raj-01 Raja undulata Undulate ray 2 80Built linking dataframe
Tags and individuals are linked by an association through three
fields that determines how such relationship is defined. These fields
are: 1. is_internal: whether the tag is internal or
external 2. is_active: whether the tag is active or not 3.
tagged_date: a datetime field following ymd_hms format,
i.e, 2024-11-04 17:35:35
tag_individual_prefix <- "tagged_individual_"
# Again, we build three random alphanumeric ids for shwocasing purposes
is_internal <- c(TRUE, TRUE, FALSE)
is_active <- rep(TRUE, 3)
tagged_date <- c(rep("2024-11-11 12:02:00", 2), "2024-11-11 13:01:17")
datasets[[length(datasets) + 1]] <- data.frame(merged_id = 1:3) %>%
dplyr::mutate(
!!paste0(tag_individual_prefix, "is_internal") := is_internal,
!!paste0(tag_individual_prefix, "is_active") := is_active,
!!paste0(tag_individual_prefix, "tagged_date") := tagged_date
)
datasets[[4]]
#> merged_id tagged_individual_is_internal tagged_individual_is_active tagged_individual_tagged_date
#> 1 1 TRUE TRUE 2024-11-11 12:02:00
#> 2 2 TRUE TRUE 2024-11-11 12:02:00
#> 3 3 FALSE TRUE 2024-11-11 13:01:17Merge all datasets into a single dataframe
Last and before sending the details to the API we need to merge all datasets.
options(width = 600)
tagged_individual_df <- datasets %>%
purrr::reduce(dplyr::full_join, by = "merged_id") %>%
dplyr::select(-merged_id)
tagged_individual_df %>%
assertr::verify(nrow(.) == 3) %>%
assertr::verify(!"merged_id" %in% names(.))
#> sensor_id sensor_sensor tag_serial_number tag_frequency tag_comm_protocol ind_individual_id ind_scientific_name ind_common_name ind_sex ind_total_length tagged_individual_is_internal tagged_individual_is_active tagged_individual_tagged_date
#> 1 11744 DEPTH vignette_90SU 69 kHz Ops MOVE-RUN-fane-01 Trisopterus luscus Faneca 1 35 TRUE TRUE 2024-11-11 12:02:00
#> 2 11722 PRESSURE vignette_90SU 69 kHz Ops MOVE-RUN-fane-01 Trisopterus luscus Faneca 1 35 TRUE TRUE 2024-11-11 12:02:00
#> 3 11775 SALINITY vignette_Hjls 69 kHz Ops MOVE-RUN-raj-01 Raja undulata Undulate ray 2 80 FALSE TRUE 2024-11-11 13:01:17Send all details to the API.
We are ready to send all the details to the API,
ret <- templo_client$add_tagged_individuals(
tagged_individual_df,
individual_prefix,
tag_prefix,
sensor_prefix,
tag_individual_prefix
)… But we need to ensure that the records were actually inserted.
options(width = 600)
api_data <- templo_client$get_tagged_individuals(end_point = "tagged_individuals/")
#> INFO [2024-11-14 17:08:22] Fetching all tagged individuals from the api... This may take a few seconds
#> INFO [2024-11-14 17:08:23] ---> Converting results to dataframe...
#> INFO [2024-11-14 17:08:26] Fetching all sensor types from the api... This may take a few seconds
#> INFO [2024-11-14 17:08:26] --->Finished
#> INFO [2024-11-14 17:08:26] --->Finished
api_data %>%
dplyr::filter(stringr::str_detect(serial_number, "vignette")) %>%
assertr::verify(nrow(.) == 3)
#> sensor_id tagged_individual_id is_internal is_active tagged_date ind_integer_id individual_id weight sex total_length tag_id serial_number comm_protocol frequency sensor_type
#> 1 11722 392 TRUE TRUE 2024-11-11T12:02:00 321 MOVE-RUN-fane-01 NA 1 35 391 vignette_90SU Ops 69 PRESSURE
#> 2 11744 392 TRUE TRUE 2024-11-11T12:02:00 321 MOVE-RUN-fane-01 NA 1 35 391 vignette_90SU Ops 69 DEPTH
#> 3 11765 394 FALSE TRUE 2024-11-11T13:01:17 331 MOVE-RUN-raj-01 NA 2 80 393 vignette_SnRU Ops 69 SALINITYDelete an individual or tag
There may be sometimes where one may have associated the wrong individual to a tag, or simply the information of a sensor is wrong. The API is equipped with individual methods that let a user update certain information at specific tables to fix this issue. However, this is a complex operation that requires familiarisation with the relationship among tables. For now, whenever a situation of such nature appears, the process to fix it consist in a) deleting the records concerning that individual or tag and b) re-inserting again as we did earlier on. This section only shows how to accomplish the deletion operation, which needs to be performed following a strict order of operations as shown below:
- Delete the linked record in
tagged_individualtable by using thetagged_individual_idfield - Delete sensor part from
sensorstable by using thesensor_idfield - Delete the tag part from
tagstable by using thetags_idfield - Lastly delete the individual from
individualstable by using theindividual_idfield
Once the four operations above are completed, we can repeat the process to insert a tagged individual as shown before. For showcasing purposes, we will delete the Raja undulata individual we inserted before.
Please notice the following:
- All chunks below have been tagged with
eval=FALSEto avoid the vignette failing again, so that means that they were run only once and never again. Bear this in mind when running this vignette - The ids used for the record deletion were based on a previous insertion. Though the sections below show the correct and actual record ids, they were updated manually to make the vignette consistent
Delete tagged_individual record
ret <- templo_client$delete_single_record(end_point = "tagged_individuals/", data = list(
tagged_individual_id = 394 # id obtained from earlier section's dataframe
))
retDelete sensor record
ret <- templo_client$delete_single_record(end_point = "sensors/", data = list(
sensor_id = 11765 # id obtained from earlier section's dataframe
))
retDelete tag record
ret <- templo_client$delete_single_record(end_point = "tags/", data = list(
tag_id = 393 # id obtained from earlier section's dataframe
))
retDelete individual record
Notice that although the field used to delete a record in the
individuals table is individual_id, the value
that needs to be passed on is the integer one and not the unique string
that identifies the individual within the table. This is caused by a
disagreement in nomenclature.
ret <- templo_client$delete_single_record(end_point = "individuals/", data = list(
individual_id = 331 # id obtained from earlier section's dataframe
))
ret