Atlas

Atlas is the map for transporting data through our RPC service ecosystem. It provides a router that routes RPC calls to the proper service locations and holds all of the definitions for the RPC requests and responses (in Google Protocol Buffer format). Combined with Active Remote, it make sending/receiving data to/from any of our services simple.

Search Mind About Atlas - https://mind.internal.mx/search?q=atlas

Installation

Ruby

Add this line to your application's Gemfile:

gem 'atlas'

And then execute:

$ bundle

Or install it yourself as:

$ gem install atlas

Go

1. Add this to go.mod in your project

gitlab.com/mxtechnologies/mx/atlas.git/v5 <latest-version>

2. Run the following commands

• git config --global url."[email protected]:".insteadOf "https://gitlab.com/"
• go env -w GOPRIVATE="gitlab.com/mxtechnologies/*"
• go mod tidy

TypeScript

Add to your project with npm, yarn, pnpm, or bun:

npm install @mxenabled/atlas
# or
bun add @mxenabled/atlas

Generated message types and service definitions are imported directly from the package. For example:

import { Account, AccountServiceDefinition } from "@mxenabled/atlas/ts/atlas/abacus/account.pb";

Service definitions (*ServiceDefinition) are emitted via ts-proto's outputServices=generic-definitions option. They are transport-agnostic descriptors — pair them with a NATS RPC client of your choosing to dispatch requests.

Doc generation

1. Install Go
2. Install the docs generator with:

go install github.com/pseudomuto/protoc-gen-doc/cmd/protoc-gen-doc@latest

3. Generate the docs with:

rake docs:compile

Usage

At this point, while it is technically possible to use Atlas's RPC functionality, the best way to use Atlas is through Active Remote. But if you insist on using it directly, here's how...

tag = ::Atlas::Abacus::Tag.new(:name => 'foo')

# To create a tag in Abacus, call service like this:
::Atlas::Abacus::TagService.client.create(tag) do |c|
  # In the event of service failure, raise the error.
  c.on_failure do |error|
    raise error.message
  end

  # In the event of service success, assign the response.
  c.on_success do |response|
    @response = response
  end
end

The on_failure and on_success callbacks are essential for handling the RPC response. Without them, it's Just Not Gonna Work™.

Development

Our RPC data format of choice is Google's Protocol Buffers and we use the protobuf gem to compile definitions into protocol buffer messages.

NOTE: Minimum supported protoc version is 3.6.0.

Contributing

See process wiki page for details on updating a gem.

1. Create your feature branch (git checkout -b my-new-feature)
2. If changing a .proto file
   • brew install go (unless you already have it)
   • brew install protobuf (unless you already have it)
   • brew install node (unless you already have it)
   • npm install (installs ts-proto and typescript for the TypeScript bindings)
   • rake compile
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

NOTE -> PYTHON:

brew install protobuf loads the latest proctoc version, which will lead to problems downstream with the analytics platform. Need to stay on "29.5" until the analytics platform can support a higher version. The protobuf runtime must be newer or equal to the version of protoc.

Releasing

To merge and publish, comment shipit --publish-version=minor/patch once the MR is approved.
Shipit will handle version bumping based on the specified flag (minor, or patch), publish the package to artifactory, merge the MR into the master branch, and create a corresponding tag.

Publishing the TypeScript package

The @mxenabled/atlas npm package is published manually. Before publishing, sync package.json to the canonical version in lib/atlas/version.rb:

bundle exec rake sync_ts_version
npm publish

Common Patterns & Tips

Adding a new version

Atlas' versioning changes much more frequently than other gems on our Platform™. To maintain some semblance of sanity, slight modifications should be shipped with shipit --publish-version=patch to bump a patch version (e.g. 0.1.0 to 0.1.1) and more significant changes like new messages or endpoints should shipped with shipit --publish-version=minor resulting in a minor bump (e.g. 0.1.1 to 0.2.0).

Adding new fields to a model

One of the most common things we do to Atlas is let it know of fields we've added. Those fields could be simple attributes or attributes backed by a database column.

For example, you've just added a migration that adds a verified_at column to the MicroDeposit model in Persona.

In Atlas, you'd then:

Update the Atlas gem

1. Open the definitions/atlas/persona/micro_deposit.proto file.
2. Add optional int64 verified_at = 24; to the message MicroDeposit { block
3. Add repeated int64 verified_at = 14; to the message MicroDepositRequest { block (if needed).
4. Run bundle exec rake compile to compile the changed .proto file.
5. Submit an MR
6. Merge that MR into master upon approval

Update other services

Once the Atlas gem is published, you now need to make sure that any services that use whatever you've added to Atlas has the latest Atlas version.

For example, if you added a new column/attribute to Persona, and you have code in Harvey that needs to use that attribute, you'll need to make sure that both Persona and Harvey have the version of Atlas that includes that attribute in its .proto files.

At the very least, you'd want to run the following in Persona and Harvey:

bundle update --conservative atlas

Alternatively, if you want to update the atlas version of one of our Go services, you would use:

go get -u gitlab.com/mxtechnologies/mx/atlas.git/v5

Typically you don't need to worry about creating a release when you're only updating the Atlas gem version in a repo. Make sure you have the latest master branch for the repo, run the above, add, commit, and push the changes.

Adding new guid type

To add a new guid type

1. Edit guids.yml
2. Add a new guid pair with resource name camelized and prefix (anywhere) like:

MountainRange: MRG

rake precompile will alphabetize the guids.yml file, and build the Ruby, Go, and TypeScript versions of the constants. This is included automatically in rake compile

Troubleshooting

Compilation Issues

1. Run bundle update to make sure you are using the latest version of mx-go-tool.
2. Run bundle exec rake compile:troubleshoot and make sure you are using the correct binaries. They should be ones coming from the mx-go-tools gem or rbenv.
   • Install the latest protobuf compiler if it was not found. macOS install or binaries
   • Make sure you are using a version of Ruby managed by Rbenv (eg. rbenv local 2.5.1) or your Ruby version manager of choice. Do not use the system version as this prevent the correct binaries from being found.
3. Make sure you are using the latest version of the protobuf compiler.
   • If you are using brew then run brew update && brew upgrade protobuf.
   • Released binaries

Newly added fields not behaving as expected

The scenario: you've added a database column to the MicroDeposit model in Persona, added the correct code to the micro_deposit.proto file, compiled the proto file, and released the gem. You've also made sure to update the version of Atlas in Harvey.

In Harvey, you can create an instance of the ActiveRemote model and see the newly added attribute, but when you assign a value to the attribute for that instance and save, it doesn't actually persist the value. Other attributes might work fine, just not yours.

Spoiler: Harvey has the newest Atlas, but Persona doesn't. Persona needs to be updated to use the latest Atlas so it can setup the methods it needs to read or write to the newly added attribute.

In short: when you add a new attribute/column/method to a model, everything that uses or could use that new attribute needs to use the latest Atlas, include the service where the attribute originates.

Deprecations

To see where deprecated fields are being used in other apps, run the following command in an app that uses Atlas. Running this will cause any spec to fail that is using a deprecated field: ATLAS_SKIP_DEPRECATED_FIELDS="true" bx rspec spec