DATA FLOWS - Revision history

Admin: /* {{Fa|fa-upload}} (IN) Manual data import */

2024-04-24T11:23:00Z

‎{{Fa|fa-upload}} (IN) Manual data import

Admin: /* {{Fa|fa-upload}} (IN) Manual data import */

2024-04-24T11:19:57Z

‎{{Fa|fa-upload}} (IN) Manual data import

Admin: /* {{Fa|fa-upload}} (IN) Manual data import */

2024-04-24T10:34:23Z

‎{{Fa|fa-upload}} (IN) Manual data import

Admin: /* {{Fa|fa-upload}} (IN) Manual data import */

2024-04-24T07:55:07Z

‎{{Fa|fa-upload}} (IN) Manual data import

Admin: /* {{Fa|fa-upload}} (IN) Manual data import */

2024-02-29T04:51:29Z

‎{{Fa|fa-upload}} (IN) Manual data import

Admin: /* {{Fa|fa-upload}} (IN) Manual data import */

2024-02-29T04:29:08Z

‎{{Fa|fa-upload}} (IN) Manual data import

Admin: /* {{Fa|fa-upload}} (IN) Manual data import */

2024-02-28T19:19:23Z

‎{{Fa|fa-upload}} (IN) Manual data import

Admin: /* Movebank to Anitra */

2024-02-28T18:51:10Z

‎Movebank to Anitra

Admin: /* Ornitela */

2024-02-28T18:49:16Z

‎Ornitela

Admin: /* Ornitela */

2024-02-28T18:46:06Z

‎Ornitela

@@ Line 55: / Line 55: @@
 : '''2) File format and structure checking'''
-::* CSV files only are accepted currently.
+:: CSV files only are accepted currently.
-::* The CSV is supposed to be well structured i.e.:
+:: The CSV is supposed to be well structured i.e.:
-::** Columns to be delimitted by a valid column separators (TAB, comma or semi-colon)
+::* Columns to be delimitted by a valid column separators (TAB, comma or semi-colon)
-::** UTF-8 data encoding to be used etc
+::* UTF-8 data encoding to be used etc
-::* First row must contain column list
+:: First row must contain column list
-::* column names must be unique and not empty
+:: column names must be unique and not empty
 :: Wrong files will display errors and would not be imported.
@@ Line 100: / Line 100: @@
 :: Note the import procedure performs append and merge data operations  (Also note no data delete occurs).
-::* For any new device (device_code) detected in the file a new Device will be created in Anitra and all the imported rows linked to it.
+::: For any new device (device_code) detected in the file a new Device will be created in Anitra and all the imported rows linked to it.
-::* For devices matching existing Anitra devices the data from the file will be merged (rows inserted or updated using data timestamp). Note a possible data loss by overwriting existing data (the same combination of device and date time already exists in the database).
+::: For devices matching existing Anitra devices the data from the file will be merged (rows inserted or updated using data timestamp). Note a possible data loss by overwriting existing data (the same combination of device and date time already exists in the database).
 :: In case no data errors are detected user can decide if new rows should be inserted and/or existing data should be updated.
@@ Line 109: / Line 109: @@
 : Note you can import simultaneously more files in one import session. Each of the files runs independently through its 5-step import flow.
-: As the import of large files can last many minutes or even hours (depending on server overload), you can close the browser tab (or even your computer) during the last import step. In case you need to return to the import screen use the "Running imports" list in the customer profile to reopen the session again<br>[[File:customer_import_04_running_imports.jpg|900px]]
+: As the import of large files can last many minutes or even hours (depending on server overload), you can close the browser tab (or even your computer) any time during the import. In case you need to return to the import screen use the "Running imports" list in the customer profile to reopen the session again<br>[[File:customer_import_04_running_imports.jpg|900px]]
 : [[IMPORT_OLD| Manual import functionality - old version (before 2024)]]

@@ Line 88: / Line 88: @@
 : '''4) Validate data'''
-:: Once the import file is succesfuly matched a data checking phase follows (i.e. rows are checked)
+:: Once the import file is successfully matched a data-checking phase follows (i.e. rows are checked)
-:: Data validation step checks the values against datatypes expected in target columns (e.g. wrong date time format, string data too long, text data in numeric columns or similar datatype conflicts etc)
+:: Data validation step checks the values against datatypes expected in target columns (e.g. wrong date time format, string data too long, text data in numeric columns, or similar datatype conflicts, etc)
-:: The rows/cells with problem data are detected and problems shown.
+:: The rows/cells with problem data are detected and problems are shown.
-:: ''Note the past option of skipping the error rows processing only non error data not currently implemented. I.e any data error in the file prevents the whole file from being imported.''<br>[[File:customer_import_03b_import_wizard_data_err.jpg|900px]]
+:: ''Note the past option of skipping the error rows processing only non-error data not currently implemented. I.e any data error in the file prevents the whole file from being imported.''<br>[[File:customer_import_03b_import_wizard_data_err.jpg|900px]]
-:: The second major task of the validation phase is to check previous existence of the rows in the database.
+:: The second major task of the validation phase is to check the previous existence of the rows in the database.
-:: Rows mathing existing records are prepared to be possibly '''updated'''.
+:: Rows matching existing records are prepared to be possibly '''updated'''.
 :: Rows not matching existing records are prepared to be '''inserted'''.
-:: Additionally, a special purpose rows are removed from the file (e.g. not processsed sensor rows in Ornitela files)
+:: Additionally, special purpose rows are removed from the file (e.g. not processed sensor rows in Ornitela files)
-: Note the import procedure performs append and merge data operations  (Also note no data delete occures)
+:: Note the import procedure performs append and merge data operations  (Also note no data delete occurs).
-:* For any new device (device_code) detected in the file a new Device will be created in Anitra and all the imported rows linked to it
+::* For any new device (device_code) detected in the file a new Device will be created in Anitra and all the imported rows linked to it.
-:* For devices matching existing Anitra devices the data from the file will be merged (rows inserted or updated using data timestamp). Note a possible data loss by overwritiong existing data (same combination of device and date time already exists in the database)
+::* For devices matching existing Anitra devices the data from the file will be merged (rows inserted or updated using data timestamp). Note a possible data loss by overwriting existing data (the same combination of device and date time already exists in the database).
-: [[IMPORT_OLD| Manual import functionality - old vereison (before 2024)]]
+:: In case no data errors are detected user can decide if new rows should be inserted and/or existing data should be updated.
 = (OUT) Anitra API data & metadata access =

@@ Line 41: / Line 41: @@
 = {{Fa|fa-upload}} (IN) Manual data import=
-: [[IMPORT_OLD| Manual import functionality - old vereison (before 2024)]]
+: Data Import dialog can be accessed in the Data management section of your Customer/Organization profile.
-: Data Import dialog can be accessed in Data management section of your Customer/Organization profile
+: An import wizard in a separate browser tab is open and automated file preprocessing (checking file type, header/columns, and actual data in the file) is run.
-: Imported data are merged under Customer/Organisation account (new devices added or data appended to existing devices)<br>[[File:customer_import_01_file_selection.jpg|900px]]
+: Messages about the progress of file checking are displayed.
-: Import dialog now accepts data in the following structures/teplates:
+: '''1) Aquiring file'''
-: 1) data mapped directly to Anitra database column names - labeled as a '''Basic format''' in the Import dialog
+: '''2) File format and structure checking'''
-:: Designed for importing custom data
+: '''3) Detect import file format'''
-:: Check our guideline and sample files while preparing your import files
+:: Import dialog now accepts data in several formats/structures
-::* [https://memos-my.sharepoint.com/:x:/g/personal/dusan_rak_anitra_cz/EW3wvkVaaFlMjlp9yFMEgT4BFu_8rAWtlCh2nOUuVrfmkQ?e=XenvK8| guideline file] - explains mandatory columns, lists correct column names and data types to be used, and gives some example values
+:: The format is automatically detected from the first file row (column header).
-::* [https://memos-my.sharepoint.com/:x:/g/personal/dusan_rak_anitra_cz/EQhkt8fmgPVDjUak8IhbxPkBceKv7jXoiF2dhp6gcJ0lMg?e=GeAxvj| template CSV file] - this is how your files should be structured. Note the file contains a header row and several sample/test rows.
+:: To pass this step successfully the column list must match one of the accepted formats i.e the mandatory columns (such as device identification, datetime, or latitude and longitude) per format to be supplied and matching
-:: For successful import make sure you use correct datatypes, column and data delimitters and the file encoding.
+:: The following formats/structures/templates are now accepted:
-: 2) for importing data in the original file structure and format generated by other tag producers (fixed custom mapping of original files to Anitra database).
+:: '''Movebank''' data (to be imported via Basic format currently)
-::* '''Ecotone CSV format'''
+:: For importing data in the original CSV file structure and format generated by some other tag producers (fixed custom mapping of original files to Anitra database) these import formats are also implemented.
-::* '''Ornitela CSV format'''
+:: '''Ecotone'''
-::: Ornitela import currently accepts muptiple versions of the Ornitela CSV file (versions 1 and 2, GPS, GPS+SENSORS etc). Note the "SENSORS" rows are skipped and not imported.
+::: Devices created using Ecotone import format are automatically labeled as device type "Ecotone" in Anitra (same as if these were created via Ecotone integration - see automatic integration section)
 :: Flows using native file formats can be useful when uploading older data files not available anymore to automated integration flow e.g. older records after having activated Ornitela SFTP data flow (which only generates new records).
-: Note this general behavior traits of the import flow
+:: Data validation step checks the values against datatypes expected in target columns (e.g. wrong date time format, string data too long, text data in numeric columns or similar datatype conflicts etc)
-:* import procedure performs append and merge data operations  i.e.
+:: The second major task of the validation phase is to check previous existence of the rows in the database.
-:** for any new device (device_code) detected in the file a new Device will be created in Anitra and all the imported rows linked to it
+:: Rows mathing existing records are prepared to be possibly '''updated'''.
-:** for devices matching existing Anitra devices the data from the file will be merged (rows inserted or updated using data timestamp, no delete occures here)
+:: Rows not matching existing records are prepared to be '''inserted'''.
-:* first step of the import flow evaluates the file structure and the data contained
+: Note the import procedure performs append and merge data operations  (Also note no data delete occures)
-:** completely wrong files will display errors and will not be imported. following type of problems will prevent file from being imported
+:* For any new device (device_code) detected in the file a new Device will be created in Anitra and all the imported rows linked to it
-:*** wrong column separators,
+:* For devices matching existing Anitra devices the data from the file will be merged (rows inserted or updated using data timestamp). Note a possible data loss by overwritiong existing data (same combination of device and date time already exists in the database)
-:* similarly as for the file structure the rows are also checked. The rows with problem data are detected, problems shown and rows skipped while importing. You can possibly notice these
+: [[IMPORT_OLD| Manual import functionality - old vereison (before 2024)]]
 = (OUT) Anitra API data & metadata access =

@@ Line 41: / Line 41: @@
 = {{Fa|fa-upload}} (IN) Manual data import=
-: Data Import dialog for organisation level data can be accessed in Data management section of your customer profile
+: [[IMPORT_OLD| Manual import functionality - old vereison (before 2024)]]
-: Imported data are merged under Customer/Organisation account (new devices added or data appended to existing devices)<br>[[File:manual_customer_import.JPG|1200px]]
 : Import dialog now accepts data in the following structures/teplates:

@@ Line 49: / Line 49: @@
 :: Designed for importing custom data
-:: Chheck our sample and guideline files while preparing your import files
+:: Check our guideline and sample files while preparing your import files
-::* [https://memos-my.sharepoint.com/:x:/g/personal/dusan_rak_anitra_cz/EW3wvkVaaFlMjlp9yFMEgT4BQP70zuQP06WetSiTjCDcQA?e=ACImZe| guideline file] - explains mandatory columns, lists correct column names and data formats to be used (sample/example values given)
+::* [https://memos-my.sharepoint.com/:x:/g/personal/dusan_rak_anitra_cz/EW3wvkVaaFlMjlp9yFMEgT4BFu_8rAWtlCh2nOUuVrfmkQ?e=XenvK8| guideline file] - explains mandatory columns, lists correct column names and data types to be used, and gives some example values
-::* template CSV file - this is how your files should be structured. Note the file contains a header row and several sample/test rows
+::* [https://memos-my.sharepoint.com/:x:/g/personal/dusan_rak_anitra_cz/EQhkt8fmgPVDjUak8IhbxPkBceKv7jXoiF2dhp6gcJ0lMg?e=GeAxvj| template CSV file] - this is how your files should be structured. Note the file contains a header row and several sample/test rows.
-[https://memos-my.sharepoint.com/:x:/g/personal/dusan_rak_anitra_cz/EW3wvkVaaFlMjlp9yFMEgT4BQP70zuQP06WetSiTjCDcQA?e=ACImZe| guideline file]
+: 2) for importing data in the original file structure and format generated by other tag producers (fixed custom mapping of original files to Anitra database).
-: 2) for importing data in the original file structure and format generated by other tag producers (fixed custom mapping of original files to Anitra database).
+::* '''Ecotone CSV format'''
-::* labeled as '''Ecotone CSV format'''
+:: Flows using native file formats can be useful when uploading older data files not available anymore to automated integration flow e.g. older records after having activated Ornitela SFTP data flow (which only generates new records).
 : Note this general behavior traits of the import flow

@@ Line 41: / Line 41: @@
 = {{Fa|fa-upload}} (IN) Manual data import=
-Data Import dialog for orgaanisation level data can be accessed in Data management section of your customer profile
+: Data Import dialog for organisation level data can be accessed in Data management section of your customer profile
-<br>[[File:manual_customer_import.JPG|1200px]]
+: Imported data are merged under Customer/Organisation account (new devices added or data appended to existing devices)<br>[[File:manual_customer_import.JPG|1200px]]
-Import dialog now accepts data in the following structure/teplates:
+: Import dialog now accepts data in the following structures/teplates:
-) data mapped directly to Anitra database column names:
+: 1) data mapped directly to Anitra database column names - labeled as a '''Basic format''' in the Import dialog
-) data in the original file structure and format of Anitra and some other tag producers (fixed custom mapping of original files to Anitra database).
+:* option of importing custom data
-* labeled as "Anitra CSV format"
+:* please download this template file listing available columns and sample values to be used while importing
-Note this general behavior traits of the import flow
+::* labeled as '''Ecotone CSV format'''
-* import procedure performs append and merge data operations  i.e.
+:: Note these flows can be useful when uploading older data files not available anymore to automated flow e.g. older records after having activated Ornitela SFTP data flow (which only generates new records).
-* first step of the import flow evaluates the file structure and the data contained
+: Note this general behavior traits of the import flow
-* similarly as for the file structure the rows are also checked. The rows with problem data are detected, problems shown and rows skipped while importing. You can possibly notice these
+:* import procedure performs append and merge data operations  i.e.
-** errors:
+:** for any new device (device_code) detected in the file a new Device will be created in Anitra and all the imported rows linked to it
-*** data in the cell not matching the expected target format (e.g. wrong date time format, string data too long, text data in numeric columns or similar datatype conflicts etc)
+:** for devices matching existing Anitra devices the data from the file will be merged (rows inserted or updated using data timestamp, no delete occures here)
-Note
+:* similarly as for the file structure the rows are also checked. The rows with problem data are detected, problems shown and rows skipped while importing. You can possibly notice these
-* any user (inculding those not involved in any Customer account) can upload data under their private user profile (ask us in case you need this feature activated for your user account)
+:** errors:
-** in this case devices created are stored in users profile only and have no relationship with any Customer/Organisation entity
+:*** data in the cell not matching the expected target format (e.g. wrong date time format, string data too long, text data in numeric columns or similar datatype conflicts etc)
-** the user who uploaded the data becomes "Device Owner"
+:*** all these rows will be skipped during the import<br>[[File:import_data_errors.jpg|1200px]]
-* for importing data in the particular customer profile (not in personal profile) make sure you chose one of your organization accounts in the "Upload in profile" selection
+:** warnings:
-** in this case data imported will be incorporated under the selected "Customer/Organisation" entity
+:*** possible data loss by overwritiong existing data (same combination of device and date time already exists in the database)
 = (OUT) Anitra API data & metadata access =

@@ Line 35: / Line 35: @@
 == Movebank to Anitra ==
-:* configuring this connector a Movebank data + Movebank metadata are connected
+: Configuring this connector a Movebank data (+ Movebank metadata) are connected
-:* Movebank API is employed for accessing the records
+: Movebank API is employed for accessing the records
-:* in this case user is supposed to enter a movebank username and password in combination with Movebank Study ID and request the data flow activation <br>[[File:Movebank2Anitra_setup_wizard.JPG|400px]]
+: In this case user is supposed to enter a movebank username and password in combination with Movebank Study ID and request the data flow activation <br>[[File:Movebank2Anitra_setup_wizard.JPG|400px]]
 = {{Fa|fa-upload}} (IN) Manual data import=

@@ Line 25: / Line 25: @@
 : Details needed: Ornitela user name and password
 : Ornitela integration is realised using "Ornitela SFTP" functionality.
-: i.e. in this case user creates, edits the data connector, and requests activation (or requests deactivation) of the data flow
+:* i.e. in this case user creates, edits the data connector, and requests activation (or requests deactivation) of the data flow
-: by requesting activation user also requests to have the SFTP configuration in his Ornitela pannel configured by us. <br>[[File:Ornitela2Anitra_setup_wizard.JPG|400px]]
+:* by requesting activation user also requests to have the SFTP configuration in his Ornitela pannel configured by us. <br>[[File:Ornitela2Anitra_setup_wizard.JPG|400px]]
 : Note other related aspects using SFTP:
 :* only new data are integrated once the data flow is activated.
-:* for importing older data (records created before the feed activation) use manual import functionality (any duplicate records are merged or skipped so you can import the same period repeatedly).
+:* for importing older data (records created before the feed activation) use the [[DATA_FLOWS#.28IN.29_Manual_data_import|Manual Data Import]] functionality (any duplicate records are merged or skipped so you can import the same period repeatedly).
 :* only the devices marked for SFTP export in the Ornitela panel will be automatically updated into Anitra
 :* make sure you also update the connector definition anytime you are changing your Ornitela user password