Facilitate ETL operations to bring non-TAK sources into a TAK Server
Local installation can take advantage of the docker-compose workflow.
docker-compose up --build
Once the database and API service have built, the server will start on port 5000.
In your webbrowser visit http://localhost:5000 to view the ETL UI
For non-aws environments, the default username & password is admin, admin.
This value can be customized via the TAK_USERNAME & TAK_PASSWORD env vars
Installation outside of the docker environment is also fairly straightforward.
In the ./api, perform the following
npm install
echo "CREATE DATABASE tak_ps_etl" | psql
npx knex migrate:latest
cd web/
npm install
npm run build
cd ..
npm run devThe ETL service assumes several pre-requisite dependencies are deployed before initial ETL deployment. The following are dependencies which need to be created:
| Name | Notes |
|---|---|
coe-vpc-<name> |
VPC & networking to place tasks in - repo |
coe-ecs-<name> |
ECS Cluster for API Service - repo |
coe-ecr-etl |
ECR Repository for storing API Images - repo |
coe-ecr-etl-tasks |
ECR Repository for storing Task Images - repo |
coe-elb-access |
Centralized ELB Logs - repo |
An AWS ACM certificate must also be generated that covers the subdomain that CloudTAK is deployed to as well
as the second level wildcard. Where in the example below CloudTAK is deployed to ie: map.example.com The second
level wildcard will be used for serving tiles, currently configured to be tiles.map.example.com
IE:
*.example.com
*.map.example.com
| Name | Notes |
|---|---|
coe-media-<name> |
Task Definitions for Media Server Support - repo |
coe-ecr-etl
Can be created using the dfpc-coe/ecr repository.
From the ecr repo:
npm install
npx deploy create etlcoe-ecr-etl-tasks
Can be created using the dfpc-coe/ecr repository.
From the ecr repo:
npm install
npx deploy create etl-tasksAn S3 bucket will be created as part of the CloudFormatiom stack that contains geospatial assets related to user files, missions, CoTs, etc. The following table is an overview of the prefixes in the bucket and their purpose
| Prefix | Description |
|---|---|
attachment/{sha256}/{file.ext} |
CoT Attachments by Data Package reported SHA |
data/{data sync id}/{file.ext} |
CloudTAK managed Data Sync file contents |
import/{UUID}/{file.ext} |
User Imports |
profile/{email}/{file.ext} |
User Files |
From the root directory, install the deploy dependencies
npm installDeployment to AWS is handled via AWS Cloudformation. The template can be found in the ./cloudformation
directory. The deployment itself is performed by Deploy which
was installed in the previous step.
The deploy tool can be run via the following
npx deployTo install it globally - view the deploy README
Deploy uses your existing AWS credentials. Ensure that your ~/.aws/credentials has an entry like:
[coe]
aws_access_key_id = <redacted>
aws_secret_access_key = <redacted>
Deployment can then be performed via the following:
npx deploy create <stack>
npx deploy update <stack>
npx deploy info <stack> --outputs
npx deploy info <stack> --parameters
Stacks can be created, deleted, cancelled, etc all via the deploy tool. For further information
information about deploy functionality run the following for help.
npx deployFurther help about a specific command can be obtained via something like:
npx deploy info --help