genericsuite-be-scripts
v1.0.8
Published
GenericSuite Scripts (backend version)
Downloads
366
Maintainers
Readme
The GenericSuite Scripts (backend version).
GenericSuite is a versatile backend solution, designed to provide a comprehensive suite of features for Python APIs. It supports various frameworks including Chalice, FastAPI, and Flask, making it adaptable to a range of projects. This repository contains the backend scripts necessary to build and deploy APIs made by the backend version of The GenericSuite.
Features
- AWS Deployment: Deployment to AWS as Lambda Function with API Gateway usig SAM (AWS Serverless Application Model).
- Local Development Environment: running with http or https, with or without Docker.
- Local DNS Server: to allow https API access with a domain name like
app.exampleapp.local
and allow access from another devices locally (e.g. smartphones) to test your App. - Self-signed SSL certificates creation: to allow local development frontend and backend environments run over secure https connections.
- Common JSON config management: to add the Git Submodule with the common JSON config directories.
- Local MongoDB Docker conntainer: used by the test site and allows to have an offline local development environment.
Getting Started
To get started with GenericSuite, follow these steps:
Initiate your project
Check The GenericSuite Getting Started guide for more details.
Install the GenericSuite Backend Scripts
npm init
npm install -D genericsuite-be-scripts
To generate sef-signed SSL certificates, office-addin-dev-certs
is required:
npm install -D office-addin-dev-certs
Prepare the Makefile
Copy the Makefile
template from node_modules/genericsuite-be-scripts
:
cp node_modules/genericsuite-be-scripts/Makefile ./Makefile
AWS SAM
Prepare the AWS/SAM templates
Create the project's scripts/aws_big_lambda
and scripts/aws
directories and copy the templates:
make init_sam
Or...
sh node_modules/genericsuite-be-scripts/scripts/aws_big_lambda/init_sam.sh
If you're going to develop with the Chalice framework:
Create the .chalice
directory and copy the .chalice/config-example.json
template:
make init_chalice
Or...
sh node_modules/genericsuite-be-scripts/scripts/aws/init_chalice.sh
Customize SAM Templates
If you need to do any customization to the samconfig.toml
:
Edit the template-samconfig.toml
file:
vi scripts/aws_big_lambda/template-samconfig.toml
# or
# code scripts/aws_big_lambda/template-samconfig.toml
Check for some customization needed.
NOTE: The deployment script big_lambdas_manager.sh will replace APP_NAME_LOWERCASE_placeholder
with the Application name defined in the APP_NAME
variable in .env
file.
Add new Endpoints to SAM Template
When you need to add new endpoints to your App:
Edit the template-sam.yml
:
vi scripts/aws_big_lambda/template-sam.yml
# or
# code scripts/aws_big_lambda/template-sam.yml
In this file is where the endpoints are defined, as well as other SAM deployment elements like the environment variables used by the AWS Lambda function. You can add new endpoints or customize it as you needed.
There's an endpoint definition template in the file node_modules/genericsuite-be-scripts/scripts/aws_big_lambda/template-sam-endpoint-entry.yml.
Be careful about elements ending with _placeholder
because they are replaced by the deployment script big_lambdas_manager.sh
with the corresponding values.
Add new Environment Variables
If you need to add additional environment variables to your App:
Edit the update_additional_envvars.sh
file:
vi scripts/aws/update_additional_envvars.sh
# or
# code scripts/aws/update_additional_envvars.sh
Add your additional environment variables replacements in scripts/aws/update_additional_envvars.sh
as:
perl -i -pe"s|ENVVAR_NAME_placeholder|${ENVVAR_NAME}|g" "${CONFIG_FILE}"
... replacing "ENVVAR_NAME" with the name of the environment variable
Add the additional environment variables to the .env
file:
ENVVAR_NAME=ENVVAR_VALUE
... replacing "ENVVAR_NAME" with the name of the environment variable and ENVVAR_VALUE with its value.
Add the additional environment variables to the scripts/aws_big_lambda/template-sam.yml
file, in the APIHandler > Properties > Environment > Variables
section. E.g.
.
.
APIHandler:
.
.
Properties:
.
.
Environment:
Variables:
ENVVAR_NAME: ENVVAR_VALUE
.
.
... replacing "ENVVAR_NAME" with the name of the environment variable and ENVVAR_VALUE with its value.
If you're using the Chalice framework, add the additional environment variables to the .chalice/config-example.json
file, in the main environment_variables
section. E.g.
{
"version": "2.0",
.
.
"environment_variables": {
.
.
"ENVVAR_NAME": "ENVVAR_NAME_placeholder"
},
"stages": {
.
.
... replacing "ENVVAR_NAME" with the name of the environment variable (in both places).
Start Development Server
To start the development server for the dev
stage and a local docker MongoDB container:
make run
To start the development server for the qa
stage and MongoDB Atlas:
make run_qa
When there are changes to the dependencies or .env
file, restart the local development server:
make restart_qa
Deploy QA
To perform a QA deployment as an AWS Lambda Function and AWS API Gateway:
make deploy_qa
Install dependencies
- Install default package categories from Pipfile.
Runs
pipenv install
. Reference: https://pipenv.pypa.io/en/latest/commands.html#install
make install
- Install both develop and default package categories from Pipfile.
Runs
pipenv install --dev
.
make install_dev
- Install from the Pipfile.lock and completely ignore Pipfile information.
Runs
pipenv install --ignore-pipfile
.
make locked_install
- Install both develop and default package categories from the Pipfile.lock and completely ignore Pipfile information.
Runs
pipenv install --dev --ignore-pipfile
.
make locked_dev
- Generates the
requirements.txt
file. Runssh scripts/aws/run_aws.sh pipfile
.
make requirements
or...
make lock_pip_file
- Clean install.
Alias that runs
make clean_rm
andmake install
.
make fresh
Cleaning
- Alias to run
make clean_rm
,make clean_temp_dir
, andmake clean_logs
.
make clean
- Remove a virtual environment created by "pipenv run".
Runs
pipenv --rm
.
make clean_rm
- Clean logs (in /logs directory).
Runs sh
scripts/clean_logs.sh
.
make clean_logs
- Clean logs, cache and temporary files.
Runs
sh scripts/aws/run_aws.sh clean
.
make clean_temp_dir
CLI Utilities
- Install development tools (pyenv, pipenv, make, and optionally: poetry, saml2aws).
Check node_modules/genericsuite-be-scripts/scripts/install_dev_tools.sh for more details about how to configure via
.env
file.
make install_tools
- Show ports in use.
Runs
sh scripts/run_lsof.sh
.
make lsof
Automated Testing
- Start the local MongoDB docker container and run the tests.
Runs
sh scripts/run_app_tests.sh
.
make test
- Execute the test without starting the local MongoDB docker container.
Runs
sh scripts/aws/run_tests.sh
.
make test_only
Linting
- Execute Prospector.
Runs
pipenv run prospector
.
make lint
- Execute MyPy.
Runs
pipenv run mypy .
.
make types
- Execute Coverage.
Runs
pipenv run coverage run -m unittest discover tests
andpipenv run coverage report
.
make coverage
- Execute Yapf Formatter and PyCodeStyle.
Runs
pipenv run yapf -i *.py **/*.py **/**/*.py
andpycodestyle
. References:
make format
- Execute Yapf (in "print the diff for the fixed source" mode) and PyCodeStyle.
Runs
pipenv run yapf --diff *.py **/*.py **/**/*.py
andpycodestyle
.
make format_check
Development Commands
- Perform a complete Lint, Type check, unit and integration test, format check, and styling before deployments.
Alias to run
make lint
,make types
,make tests
,make format_check
, andmake pycodestyle
.
make qa
- Start the local MongoDB docker container (used for testing and
dev
stage run). Runssh scripts/mongo/run_mongo_docker.sh run
.
make mongo_docker
- Stop the local MongoDB docker container.
Runs
sh scripts/mongo/run_mongo_docker.sh down
make mongo_docker_down
Chalice Specific Commands
- Set parameters on
.chalice/config.json
as the production stage. Runssh scripts/aws/set_chalice_cnf.sh prod
.
make config
- Set parameters on
.chalice/config.json
with no specific stage. Runssh scripts/aws/set_chalice_cnf.sh
.
make config_dev
- Set parameters on
.chalice/config.json
as the Development stage. Runssh scripts/aws/set_chalice_cnf.sh mongo_docker
.
make config_local
- Set parameters on
.chalice/config.json
as the QA stage with CORS specific variables replacement, to allow use the QA live database from the local development environment. Runssh scripts/aws/set_chalice_cnf.sh qa
. References:APP_CORS_ORIGIN_QA_CLOUD
andAPP_CORS_ORIGIN_QA_LOCAL
in the .env.example file.
make config_qa
- Set parameters on
.chalice/config.json
to prepare the QA deployment. Runssh scripts/aws/set_chalice_cnf.sh qa deploy
,
make config_qa_for_deployment
- Set parameters on
.chalice/config.json
as the Staging stage. Runssh scripts/aws/set_chalice_cnf.sh staging
.
make config_staging
- Create the AWS Stack via Chalice command.
Runs
sh scripts/aws/run_aws.sh create_stack
.
make build
- Generates the
requirements.txt
. Runssh scripts/aws/run_aws.sh pipfile
.
make build_local
- Describe the AWS stack with the Chalice command.
Runs
sh scripts/aws/run_aws.sh describe_stack
.
make build_check
- Alias to run
make unbuild_qa
.
make unbuild
- Delete the Chalice QA App.
Runs
sh scripts/aws/run_aws.sh delete_app qa
.
make unbuild_qa
- Delete the Chalice Staging App.
Runs
sh scripts/aws/run_aws.sh delete_app staging
make unbuild_staging
- Delete the AWS stack created by Chalice.
Runs
sh scripts/aws/run_aws.sh delete_stack
make delete_stack
AWS S3 and other
- Create a default
${HOME}/.aws/config
AWS configuration. Runssh scripts/aws/create_aws_config.sh
.
make create_aws_config
- Create the Development S3 buckets.
Runs
sh scripts/aws/create_chatbot_s3_bucket.sh dev
.
make create_s3_bucket_dev
- Create the QA S3 buckets.
Runs
sh scripts/aws/create_chatbot_s3_bucket.sh qa
.
make create_s3_bucket_qa
- Create the Staging S3 buckets.
Runs
sh scripts/aws/create_chatbot_s3_bucket.sh staging
.
make create_s3_bucket_staging
- Create the Production S3 buckets.
Runs
sh scripts/aws/create_chatbot_s3_bucket.sh prod
.
make create_s3_bucket_prod
Deployment
- Deploy the App on QA.
Runs
make create_s3_bucket_qa
,sh scripts/aws_big_lambda/big_lambdas_manager.sh sam_deploy qa
make deploy_qa
- Validate the SAM deployment templates on QA.
Runs
make create_s3_bucket_qa
,sh scripts/aws_big_lambda/big_lambdas_manager.sh sam_validate qa
make deploy_validate_qa
- Create the deployment QA package only.
Useful to check the package size and test the image by a local Docker run.
Runs
make create_s3_bucket_qa
,sh scripts/aws_big_lambda/big_lambdas_manager.sh package qa
.
make deploy_package_qa
- Deploy the App on Staging.
Runs
make create_s3_bucket_staging
,sh scripts/aws_big_lambda/big_lambdas_manager.sh sam_deploy staging
make deploy_staging
- Deploy the App on Production.
Runs
make create_s3_bucket_prod
,sh scripts/aws_big_lambda/big_lambdas_manager.sh sam_deploy prod
make deploy_prod
- Alias to run
make deploy_qa
.
make deploy
Application Specific Commands
- Execute the App locally using the development database, asking to run it over
http
orhttps
. Runsmake config_qa
,make clean_logs
, andsh scripts/aws/run_aws.sh run_local
. [???]
make run
- Execute the App locally using the QA database, asking to run it over
http
orhttps
. Runsmake config_qa
,make clean_logs
, andsh scripts/aws/run_aws.sh run_local qa
.
make run_qa
- Runs
make config_qa
,make clean_logs
, andsh scripts/secure_local_server/run.sh "down" ""
Stop and destroy the App local Docker container (for any running stage).
make down_qa
- Restart the App local Docker container running over QA.
Runs
make config_qa
,make clean_logs
,sh scripts/secure_local_server/run.sh "down" ""
andsh scripts/aws/run_aws.sh run_local qa
.
make restart_qa
- Execute the App locally using the development database (in a local Docker container), asking to execute it over
http
orhttps
. Runsmake config_local
,make clean_logs
,sh scripts/aws/run_aws.sh run_local dev
. [???]
make run_local_docker
- Execute
chalice local --port \$PORT --stage PROD
Runsmake config
,make clean_logs
,sh scripts/aws/run_aws.sh run
.
make run_prod
Common JSON config
- Add the Git Submodule with the common JSON config directories.
Runs
sh scripts/add_github_submodules.sh
.
make add_submodules
Local DNS Server
- Start the local DNS Server.
Runs
sh scripts/dns/run_local_dns.sh
make local_dns
- Runs
sh scripts/dns/run_local_dns.sh restart
to restart the local DNS Server.
make local_dns_restart
- Restart and rebuild the local DNS Server configuration when the local IP or any DNS parameters has been changed.
Runs
sh scripts/dns/run_local_dns.sh rebuild
make local_dns_rebuild
- Stop and destroy the local DNS Server.
Runs
sh scripts/dns/run_local_dns.sh down
.
make local_dns_down
- Test the local DNS Server.
Runs
sh scripts/dns/run_local_dns.sh test
.
make local_dns_test
Self-signed local SSL certificates
- Create the self-signed local SSL certificates (required to run the local development frontend and backend over https).
Runs
sh scripts/local_ssl_certs_creation.sh
.
make create_ssl_certs_only
- Copy the self-signed local SSL certificates to the frontend directory/local repository.
Runs
sh scripts/local_ssl_certs_copy.sh
.
make copy_ssl_certs
- Alias to run
make create_ssl_certs_only
andmake copy_ssl_certs
.
make create_ssl_certs
NPM library scripts
- Update the package.json file with the version and all other parameters except dependencies.
Runs
npm install --package-lock-only
.
make lock
- Test the publish to NPMJS without actually publishing.
Runs
sh scripts/npm_publish.sh pre-publish
.
make pre-publish
- Publish the scripts library to NPMJS.
Runs
sh scripts/npm_publish.sh publish
. Requirements:
make publish
Pypi library scripts
- Build 'dist' directory needed for the Pypi publish.
Runs
poetry lock --no-update
,rm -rf dist
andpython3 -m build
. Requirements:
make pypi-build
- Pypi Test publish.
Runs
make pypi-build
, andpython3 -m twine upload --repository testpypi dist/*
. Requirements:
make pypi-publish-test
- Pypi Production publish
Runs
make pypi-build
, andpython3 -m twine upload dist/*
. Requirements:
make pypi-publish
Troubleshooting
- If you get the error
Warning: Python >=3.9,<4.0 was not found on your system...
doingmake install
:
$ make install
pipenv install
Warning: Python >=3.9,<4.0 was not found on your system...
You can specify specific versions of Python with:
$ pipenv --python path/to/python
make: *** [install] Error 1
Fix it with these commands:
# Set the project Python version with pyenv
pyenv local 3.11
# Set the Python path with pipenv
pipenv --python ${HOME}/.pyenv/shims/python
And repeat make install
- If you get the warning
This version of npm is compatible with lockfileVersion@1...
doingmake install
:
npm install
npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but package-lock.json was generated for lockfileVersion@3. I'll try to do my best with it!
It's because you're using an old Node version. To solve it:
nvm node 18
And repeat make install
If you get
APP_NAME not set
message doing anymake run
, it's because the.env
file must be created or reviewed. Check the GenericsSuite (backend version) Configuration or GenericsSuite AI (backend version) ConfigurationIf you get CORS errors in the frontend and backend communication:
- To make both use
localhost
andhttp
, change these variables in the.env
file:
- To make both use
# Frontend .env file:
APP_LOCAL_DOMAIN_NAME=localhost
# Backend .env file:
APP_CORS_ORIGIN_QA_LOCAL=http://localhost:3000
And make run
both frontend and backend with the http
option.
- To make both use the local DNS server and
https
, change these variables in the.env
file:
# Frontend .env file:
APP_LOCAL_DOMAIN_NAME=app.exampleapp.local
# Backend .env file:
APP_CORS_ORIGIN_QA_LOCAL=https://app.exampleapp.local:3000
NOTE: replace exampleapp
with your App name, all lowercased.
And make run
both frontend and backend with the https
option.
- If the local DNS Server seems to be unreachable or not working:
Restart the local backend server:
make local_dns_restart
If the local IP changes, make sure to:
- Run
make local_dns_rebuild
. - Copy the
IP address
reported by the previous command.E.g.Local DNS domain 'app.exampleapp.local' is pointing to IP address '192.168.1.158'
. - Run
make restart_qa
- Add the
IP address
to the DNS Servers in your computer'sNetwork > DNS servers
settings. The new DNS ServerIP address
must be the first one in the list of DNS servers. - Restart the computer's WiFi or LAN network connection.
License
GenericSuite is open-sourced software licensed under the ISC license.
Credits
This project is developed and maintained by Carlos J. Ramirez. For more information or to contribute to the project, visit The GenericSuite Scripts (backend version) on GitHub.
Happy Coding!