Head of Platform and Data Engineering at Cardo AI. I build data platforms on Kubernetes and AWS, mostly with Trino, Spark, Airflow, Kafka, Iceberg and Delta Lake. Before that I did a PhD and a postdoc in Information Engineering at the University of Padua, working on protein docking, HPC and clinical machine learning.
What I work on
- Trino at scale on EKS: spot workers, Karpenter node pools on local NVMe, KEDA autoscaling, query retries and spill to disk, Trino Gateway
- Lakehouses on S3 with Iceberg, Delta Lake and the AWS Glue Data Catalog, fed by Postgres CDC through Kafka and Debezium
- Multi-account AWS with Terraform, GitOps with ArgoCD, observability with the Grafana LGTM stack
Open source
I contribute upstream to the projects I run in production:
- trinodb/charts: 17 merged PRs, including KEDA worker autoscaling, graceful worker shutdown, NetworkPolicy, the JMX exporter, and PDBs, lifecycle hooks and rollout strategies for Trino Gateway. Nominated as a Trino Champion in 2024 for this work.
- trinodb/trino: Delta Lake connector fixes (transaction conflict retries,
_delta_logcaching) and a SQL routine compiler fix - apache/airflow: AWS STS region support for the HashiCorp Vault provider
Projects
| Project | What it does |
|---|---|
| kafka2delta | Streams Kafka topics into Delta Lake tables. On PyPI. |
| nodereaper | Removes empty and unhealthy Kubernetes nodes when autoscalers fail to scale down |
| gluecat | Spark catalog plugin for the AWS Glue Data Catalog: one Glue catalog ID and IAM role per Spark catalog, Delta Lake tables, no Hive client. On Maven Central. |
| aws-glue-data-catalog-spark-client | Lets Apache Spark, including Spark 4, use the AWS Glue Data Catalog as its metastore |
| spark-with-glue | Apache Spark Docker image with Glue metastore support built in |
| spark-images | Apache Spark 3.5 to 4.2 images for Python 3.11 to 3.15, amd64 and arm64, on GHCR |
| charts | Helm charts for Spark Connect, Spark History Server, Trino, Karpenter and Kyverno |
| pysparkonk8s | Airflow provider that runs @task.pyspark tasks as Spark drivers on Kubernetes, with executor pods created and removed per task. On PyPI. |
| wkNNMI | R package on CRAN for mutual-information-weighted k-NN imputation |
| baggage-conveyor-assignment | Assigns arriving flights to airport baggage reclaim conveyors with a mixed-integer model (OR-Tools) |
Talks and writing
- Trino Summit 2024: Optimizing Trino on Kubernetes (slides)
Articles on Medium:
- Taming noisy neighbours in a multi-tenant Trino cluster
- Trino on Kubernetes, two years after my Trino Summit talk
- Running Trino workers on spot instances
- Upgrading Trino on Kubernetes with blue/green clusters
- Blocking partial clears in Airflow 2 and 3
- Free TLS for internal services on AWS
- Enabling AWS Glue support in Apache Spark 4
- Centralized AWS CloudWatch log collection to S3
- Exploring the Machine Learning capabilities of Trino
- Testing Apache Airflow DAGs locally with Testcontainers and LocalStack
- Building a custom Apache Spark Docker image with AWS Glue Data Catalog support as metastore
- Performing Delta Table operations in PySpark with Spark Connect
Research
400+ citations, h-index 10. Papers in Bioinformatics, BMC Bioinformatics, Journal of Neurology, Machine Learning, BMC Medical Informatics and Decision Making, The Journal of Supercomputing and The International Journal of High Performance Computing Applications, plus a European patent application on predicting ALS progression (grant intended). Code for protein surfaces and interfaces: VoxSurfMPI, VoxMeshSurfOpenMP, PoCavEDT, PPIprediction, AntibodyInterfacePrediction, HSS-PPI. Clinical data imputation: icu-lab-imputation, als-registry-imputation. Metagenomics: sbv-improver-medic, one of the three winning entries of the sbv IMPROVER MEDIC challenge.
Elsewhere: Resume 路 LinkedIn 路 Medium 路 Google Scholar 路 ORCID