xg2xg README

The project's README from github.com/jhuangtw/xg2xg. As one markdown file: README.md.

A handy lookup table of similar technology and services to help ex-googlers survive the real world :) pull-requests are very welcomed. Please do not list any confidential projects!

For a working example of (some) of these technologies integrated together, see: https://github.com/google/startup-os

See also: System Design Primer, The Hadoop Ecosystem Table, Map AWS services to GCP products, Database of Databases, Google Open Source Glossary

Technology

Core Technology

Google Internal Google External Open Source / Real-World
MapReduce Apache Hadoop, Spark
Flume DataFlow Apache Beam
Protocol Buffer Protobuf, FlatBuffers Cap'n Proto, Thrift, Avro, Amazon Ion, CBOR, kryo
Stubby gRPC Thrift, Bolt
Chubby Apache Zookeeper, etcd, HashiCorp Consul
Goops / PubSub Cloud Pub/Sub Apache Kafka, Apache Pulsar, Facebook LogDevice
//base abseil

Infrastructure

Google Internal Google External Open Source / Real-World
Borg Kubernetes Apache Mesos, Apache Aurora, HashiCorp Nomad
GSLB Cloud Load Balancing - Internal AWS ELB, Istio, linkerd
OnePlatform API Gateway Swagger
GFE, Maglev, uberproxy Cloud Load Balancing - HTTPS / External envoy, AWS ALB, HAProxy, nginx, F5
uberproxy (sso proxy) Identity-Aware Proxy buzzfeed-sso, Pomerium, Envoy Gateway
Zanzibar Zanzibar Research Paper SpiceDB/authzed, Ory Keto, topaz, Opal, (iam)Keycloak, Warrant, OpenFGA
data center hardware open compute
Jupiter, Starblaze
B4, Stargate, TE
USPS, Andromeda
ESDN Faucet
broccoli man

Storage

Google Internal Google External Open Source / Real-World
GFS/Colossus HDFS, Ceph, GlusterFS, MooseFS
Bigtable Cloud Bigtable PrestoDB, Cassandra, HBase, Accumulo, DynamoDB, ScyllaDB
Spanner Cloud Spanner Vitess, CockroachDB, TiDB
ColumnIO / Capacitor BigQuery storage Apache Parquet, ORC, Apache Iceberg
sstable levelDB RocksDB, pebble
zippy Snappy lz4
RecordIO Riegeli, TFRecords, & in OR-Tools, szl stuffed-record-stream

Services

Google Internal Google External Open Source SaaS
Dremel BigQuery Apache Drill, Presto, Spark(sort-of), AWS Athena, Redshift Spectrum, Snowflake
Dremel UI Redash, Metabase, Apache Superset
Search (Mustang, Alexandria) Elasticsearch, OpenSearch, Solr, Lucene algolia
pubsub pubsub NATS.io, RabbitMQ, PubNub AWS SQS/SNS, AWS AppSync
MillWheel Cloud Dataflow Apache Flink, beam
Colab Colaboratory Jupyter Observable
PLX Google Data Studio Mode
Monarch paper
Napa paper
MakerSuite MakerSuite

DevOps

Google Internal Google External Real-World
Assimilator Dominator
Blaze Bazel Buck, Pants, please.build, Blade, recc, EngFlow, BuildBuddy, flare.build
Oncall PagerDuty, OpsGenie, VictorOps
varz/borgmon/monarch Cloud Monitoring Datadog, Prometheus, M3, librato, newrelic, skylight, scout, Scotty/tricorder, netdata, bosun, also this and this
Viceroy Cloud Monitoring Grafana
Exception/Error Tracking (thirdeye) Sentry.io, Raygun.io, Rollbar, Honeybadger, Airbrake, OverOps, ELK stack
styleguides google styleguides PEP-8, HoundCI auto-style-reviewer
Sponge EngFlow
Blaze/Forge/TAP/BuildCop Cloud Build buildkite, CircleCI, Drone, EngFlow, github, gitlabCI, jenkins, travis
Forge/ObjFS EngFlow, flare.build
Sandman(test env)/Guitar
Sisyphus / Rapid Spinnaker, lambdaCD, screwdriver.cd, CodeShip, shipit-engine, GoCD, AWS CodeDeploy, Capistrano, Fabric, ConcourseCI, samson
MPM Docker, OCI
borgcfg / gcl, Prodspec + Annealing Jsonnet, Cue AWS Cloudformation, Puppet, Chef, Salt, Ansible, Terraform, kubecfg, pulumi, Nix, Pkl
logging, analog StackDriver logstash, fluentd, PaperTrail, cernan, loki
CodeSearch, Grimoire Zoekt kythe Code Search (for Google open source code only, with separate UI for Android and Chromium. Go CLI source.) Sourcegraph, OpenGrok, livegrep
Critique, Gerrit, Mondrian etc. Gerrit Reviewable , Phabricator
cider Eclipse Che, Cloud9, gitpod.io, Coder, Code-Server (VSCode in a Tab), DevZero
buganizer Google Issue Tracker JIRA, bugzilla, github issues, Linear
Bugjuggler SnoozeThis
ToTT Google Test Blog Increment
Copybara / MOE Copybara, MOE
workflow/dependency management Luigi, Airflow, digdag, Pachyderm, Dask
ErrorProne ErrorProne SpotBugs, FindBugs
Dapper stackdriver trace zipkin, OpenTelemetry, jaeger, LightStep, Honeycomb
C++ Tips of the Week Abseil C++ Tips of the Week
DiRT ChaosMonkey, aws fis
Rosie microplane, silver-platter
API Improvements Proposals AIP
g4 {fix, submit} Trunk.io, Graphite
probers cloudprober
GWP paper Parca, Polar Signals

Security

Google Internal Google External Open Source
prodaccess/LOAS Keymaster
prod secrets/identity management chamber, knox, SPIFFE

IT / Operations / Misc

Google Internal Real-World
software engineering at google 1, 2 Software Engineering at Google: Lessons Learned from Programming Over Time
SRE @ google
valentine Vault, 1Password, Lastpass, pass, keeper
OWNERS files in repo github CODEOWNERS
snippets Khan/snippets
SnipIt recordit, CloudApp, dropbox screenshots, Snippyly
gpaste pastee, pastebin
stuff (SaaS IT management) productiv, intello, zylo
stuff (Device Management) jamf
device security monitoring Red Canary
beyondcorp beyondcorp
go/ links golinks, go, Goat, trotto, go-shorten
google3 philosophy innersource, monorepo, YouTube talk
doing code review code review
safely sharing 1-time secrets sendsecure.ly, croc, onetimesecret, privatebin
messaging mattermost, Slack, gchat
tech talks TechTalks @ Google
g3doc writethedocs, docs-as-code
GUTS spoke, freshservice
MOMA glean

PeopleOps / Culture

Google Internal Real-World
OKR CultureAmp OKR, Lattice Goals, Ally, workboard
HRIS/ERP Namely, BambooHR, Workday, Paylocity
peer bonus bonus.ly
kudos heytaco, slack ++ bot
perks fond
talks Talks @ Google
dory slido, dory(app)
edge, people ops training LifeLabs
googlegeist Culture Amp, humu, tinypulse, peakon
Meng Search Inside Yourself
Lazlo Work Rules
Claire Stapleton Tech Support - existential advice for the modern tech worker
books about google How Google Works, In The Plex, Software Engineering at Google

also check out xoogler.co, which organizes events, slack channels etc

disclaimer: I'm not affiliated with any of the technologies/products mentioned above.

disclaimer: I left Google a number of years ago so some of the naming might be dated.

xg2xg (jhuangtw/xg2xg)

Searching Database…

15.7K ·GitHub
这是做什么的?一份速查表式的对照清单,把 Google 内部的技术和服务对应到开源、云或 SaaS 上的同类替代品,由一位前 Google 员工制作,帮离开 Google 的人在真实世界里站稳脚跟。
它是一个软件,还是只是用来看的资料?它不是软件——整个项目就是一个 README,由 markdown 表格组成,分成核心技术、基础设施、存储、服务、DevOps、安全、IT/运维和人事(PeopleOps)等章节。
它能真正教会我用这些工具,还是只报个名字?它只是在表格里列出名字——没有教程、安装指南或对比评测。想深入了解,它会给你指路,比如 System Design Primer 和 Hadoop Ecosystem Table。
我没在 Google 工作过,能看懂吗?它是写给前 Google 员工的,这些人熟悉 Borg、Chubby 之类的内部名称,却不知道对应的开源产品。不过,任何好奇哪些开源或云工具扮演同样角色的人也照样能用它。
有哪些我一看就懂的对应例子?MapReduce 对应 Hadoop 和 Spark,Chubby 对应 ZooKeeper/etcd/Consul,Bigtable 对应 Cassandra/HBase/DynamoDB,Stubby 对应 gRPC,Spanner 对应 Vitess 或 CockroachDB——都是熟悉的基础设施,只是顶着 Google 的代号,每行还列出好几个备选。
它连办公室日常也覆盖吗?是的——除了基础设施,还有 IT/运维和人事(PeopleOps)章节:go/ 链接、一次性密钥分享、Slack 或 Mattermost 这类即时通讯工具、OKR 平台、通过 bonus.ly 发的同事奖励,还有《How Google Works》这类书,另外还指路了 xoogler.co 社区。
Creator
jhuangtw
Size
0.5 MB
Stars
15.7K
Updated
CommitJan 12
Age
10y
Commits · 90 Days