HACKER DOUBLE SUMMER 2022 GUIDES — Part Six: SCaLE 19X

Ana McTaggart, Michael Hackett, Federico Lucifredi

Topic:

Observability

Ftrace has been the official tracer of the Linux kernel since its introduction in 2009. Its infrastructure is used by other observability features within the kernel including perf and eBPF. The interface to ftrace is very simple and uses reads and writes to the tracefs file system, where most features can be executed from the command line. This interaction is not always that easy to use from within a program. Keeping track of the files to write to and read from requires an understanding of the tracing framework that is not intuitive.

New libraries have been created to facilitate accessing and controlling the ftrace infrastructure file system. Introducing libtracefs, a C library which has a more intuitive interface for enabling and reading trace events. There is also libtraceevent which parses the binary tracing data into human readable format, and a growing libtracecmd that creates and reads a trace.dat file that can be used to transfer saved traces across machines and to even view in visual applications like KernelShark. There is also work to include theses libraries in a python tool called trace-cruncher that will allow your python applications to have the same level of control.

This workshop will describe the interfaces to theses libraries and give examples to allow you to start writing code that enables Linux kernel tracing, and how to make useful use of it.

Pre Requisites

Basic understanding of C
Pre-downloaded/cloned copies of the following repositories and confirm you can build / install them with make; sudo make install

Pre-downloaded/cloned copies of the following repositories and confirm you can build / install them with make libs; sudo make install_libs

https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/

Room:

Marina

Time:

Thursday, July 28, 2022–14:00 to 17:00

FRIDAY, JULY 29TH

Data Security and Storage Hardening in Rook and Ceph

Audience:

Topic:

Cloud Native

We explore the security model exposed by Rook with Ceph, the leading software-defined storage platform of the Open Source world. Digging increasingly deeper in the stack, we examine hardening options for Ceph storage appropriate for a variety of threat profiles. Options include defining a threat model, limiting the blast radius of an attack by implementing separate security zones, the use of encryption at rest and in-flight and FIPS 140–2 validated ciphers, hardened builds and default configuration, as well as user access controls and key management. Data retention and secure deletion are also addressed. The very process of containerization creates additional security benefits with lightweight separation of domains. Rook makes the process of applying hardening options easier, as this becomes a matter of simply modifying a .yaml file with the appropriate security context upon creation, making it a snap to apply the standard hardening options of Ceph to a container-based storage system.

Room:

La Jolla

Time:

Friday, July 29, 2022–13:00 to 14:00

Hardening against Kubernetes Hacks

Eric Smalling

Topic:

Cloud Native

Misconfigurations in your Kubernetes deployments can create unforeseen security vulnerabilities that can give bad actors leverage to exploit containers, nodes or even the entire control plane of your cluster. In this talk I’ll show how easy it can be to break into a cluster and why using tools to find issues and enforce governance around them can make your clusters a less attractive target.

Room:

La Jolla

Time:

Friday, July 29, 2022–14:00 to 15:00

Forward to the Past: Point in time recovery for MySQL

Pep Pla

Audience:

Topic:

MySQL

Disasters happen. Somebody runs a process that renders the data in your database unusable, inconsistent, or just deleted. Wouldn’t it be great to be able to recover your database to the state just before the disaster happened?

In this talk we will learn the basics of Point in time recovery for MySQL:

- Disasters happen.

- You need three things.

- I know what you did last transaction: the binary logs.

- I need some backup.

- The quest for the third thing: find the last transaction before the disaster.

- Putting all together the traditional way.

- Make it faster using replication.

- What you can’t recover, or at least you can’t easily recover: the transactions executed after the disaster.

Room:

Carmel

Time:

Friday, July 29, 2022–13:00 to 14:00

Workshop: Getting Started with FreeBSD

Roller Angel

Audience:

You’ll leave the workshop with the knowledge to install and administer a FreeBSD Operating System and know where to go to learn even more about using FreeBSD.

Workshop Requirements: Bring a laptop computer with VirtualBox installed.

Room:

Century C

Time:

Friday, July 29, 2022–10:00 to 17:00

State of Open Source in Government

Amanda Brock

Audience:

Topic:

Open Government

Open Source Software is being curated to form our national critical infrastructure across Governments and nations. We’ll consider how open source software got to this position, challenges and risks faced, and the needs of curation and stewardship to build critical infrastructure in 2022 and beyond.

Room:

Los Angeles C

Time:

Friday, July 29, 2022–11:00 to 12:00

Technopolice: tools for fighting back surveillance cities

axel simon

Audience:

Beginner

Topic:

Open Government

In 2019, the French non-profit La Quadrature du Net launched Technopolice: a collaborative campaign to raise awareness, document, investigate and fight back against so-called “smart cities”: really, the application to the physical world of the privacy-invading data-collection techniques first honed online. Facial recognition, “suspicious” sounds and behaviour, gait recognition, all the way to predictive policing (which has been described as automated racial profiling) all these technologies are sold by for-profit companies to security-obsessed local governments eager to answer the sirens’ call.

In ways similar to open source projects, the campaign provides tools to help anyone participate and reclaim their city, in a decentralised approach. We will look at the success the campaign has had, how people not previously involved in open source or activism have picked up the tools and used them to organise locally, and what has worked and what hasn’t.

Room:

Los Angeles C

Time:

Friday, July 29, 2022–13:30 to 14:30

Building SCaLE’s OpenWrt Conference WiFi

Robert Hernandez

Audience:

Advanced

Topic:

This is an opportunity to provide a glimpse into how the SCaLE Tech Team builds, tests, deploys, and manages SCaLE’s conference WiFi utilizing OpenWrt. The team has gone to great lengths to automate the entirety of the conference network and wireless deployment via it’s open source codebase. This includes custom OpenWrt images that work across numerous embedded CPU architectures and AP hardware, build frameworks, test frameworks, and advanced provisioning technics.

We’ll cover topics including:

— Why SCaLE chose OpenWrt to provide WiFi for the conference
— The history of OpenWrt @ SCaLE.
— How the SCaLE Tech Team builds its OpenWrt images and what they included
— The flashing of the OpenWrt image onto the actual hardware. (No paperclips required)
— Automated OpenWrt image testing for fast feedback.
— Results of this automation and its impact on the development feedback loop.
— The mechanisms that enable controlling 100+ AP deployment across the conference without a centralized controller.
— Upcoming plans for SCaLE 20x WiFi.

Room:

Carmel

Time:

Saturday, July 30, 2022–11:30 to 12:30

Your bug tracker and you

Ben Cotton

Audience:

Topic:

Your project uses a bug tracker to track bugs, but are you learning from it? This talk covers how to contigure your bug tracker to capture the right information. Attendees will learn how to triage and prioritize bugs. They will learn about processes for closing bugs, and how different closure types require different processes. The talk wraps up with a discussion of bug analysis — how to filter data, what questions to ask, and what questions to not ask, using analysis of Fedora Linux bugs as an example.

Room:

Carmel

Time:

Saturday, July 30, 2022–15:00 to 16:00

Educating Underserved Populations Through The Magic of Open Source

Patrick Swartz

Audience:

Topic:

Half of the world -four billion people — cannot reliably use the Internet. It mostly boils down to:

Lack of Infrastructure: In many remote or rural areas or refugee camps internet is often slow, unreliable, or not available at all. •
Cost: many people with low or no income simply cannot afford to pay for data.

The Kiwix project can help address these barriers by making affordable educational content accessible to people without online access.

Kiwix can compress entire websites to a single file small enough to share via mobile phones, SBC computers such as a Raspberry Pi, or low-bandwidth hotspots.

After an introduction to Kiwix, we will walk through the process of taking the simple Kiwix container image, writing a Helm chart to make it easily deployable on Kubernetes, and then deploy it out as an easily maintainable application via Rancher and K3s.

Room:

Carmel

Time:

Saturday, July 30, 2022–18:00 to 19:00

Useful Uses of chroot/proot

Simon Elmir

Audience:

Systems and Infrastructure

Topic:

Boot from a LiveUSB and rescue your local install, run Debian programs on your Android phone, and update the (ARM) OS on your Raspberry Pi’s SD card from your (x86_64) Linux desktop.

— An introduction to chroot, proot, and the internals of containerization.

Room:

Century AB

Time:

Saturday, July 30, 2022–16:30 to 17:30

Using open source tools to secure your local network

Michal Hrusecky

Audience:

Beginner

Topic:

We are making open source routers. Being open source has plenty of advantages. One of them is that we can easily integrate other open source projects. In the presentation I would like to demonstrate what cool projects do we use. What they do, how are we integrating and simplifying their use. And also what other tools we developed to make it easier for people to secure their local network better. Take it as inspiration to what you can do if you have a full control over your router and if it is running GNU/Linux. As always, there is a room for improvement, but the beauty of open source communities is that we share our solutions, inspire each other and collaborate to come up with something even better. I hope that I’ll inspire you to try few tools that you haven’t tried yet and maybe join some of our efforts.

Room:

Century CD

Time:

Saturday, July 30, 2022–13:30 to 14:30

Confidential Computing: why it HAS to be open source

Mike Bursell

Audience:

Topic:

Confidential Computing is the use of hardware-based TEEs (e.g. Intel’s SGX and AMD’s SEV) to protect data and applications in use: that means that you can run workloads on a compromised or malicious system while still be assured that their confidentiality and integrity is protected. Setting this all up and deploying it is complex and has lots of moving parts. This session will discuss the core components, and look at the impact of who’s running them, who supplied them, and whether they’re open source. It will use the Enarx project (https://enarx.dev) as an example open source project to show what choices can be made to prioritise security and the importance of openness (in not just the code, but it’s development) to the project and its success.

Room:

Century CD

Time:

Saturday, July 30, 2022–15:00 to 16:00

End of Support But Not End of Use: Security Implications of Unsupported Operating Systems

John Sicklick

Audience:

Topic:

The Ubuntu operating system has increased its commercial foothold in recent years as an open-source alternative to Windows and Red Hat Enterprise Linux. As versions of Ubuntu reach End of Support, companies are faced with the choices of migrating to a supported version of Ubuntu, continuing to use an unsupported version, or paying for Extended Security Maintenance to continue to receive security updates.

What is the level of risk associated with the continued use of a version of Ubuntu that has reached End of Support? Is it necessary to pay for Extended Security Maintenance, or can other mitigations be implemented as an alternative to applying security updates? Understanding the level of risk associated with legacy operating systems will support the decision-making process when prioritizing budgets and schedules.

This presenter will discuss the results of research performed during a graduate program. Two separate versions of Ubuntu which have reached End of Support, 14.04 and 16.04, were evaluated to compare vulnerabilities present in baseline operating systems with operating systems receiving updates through Extended Security Maintenance. This evaluation included identification of vulnerabilities using commercial security scanners, such as Tenable Nessus and Rapid7 Nexpose. A select number of identified vulnerabilities were then further evaluated using Kali Linux and Metasploit to demonstrate whether these vulnerabilities could in fact be exploited.

Room:

Century CD

Time:

Saturday, July 30, 2022–16:30 to 17:30

Live Patching: a Down in the Trenches View

Sarah Newman

Audience:

Topic:

Live patching is is an interesting answer to the question of how to minimize downtime, and sometimes risk, when performing a security update compared to other options. Typically live patching is only done to operating systems and hypervisors. Everyone should use live patches from their software vendor rather than making their own. But how does live patching work under the hood? How are they made? And if you don’t have a software vendor, is it an impossibly difficult task?

Trampolines and editing a function in place are two different methods of performing a live patch. Trampolines have fewer constraints and can use affordances present for probes or tracing.

Blindly generating a live patch may result in a change that is ineffective or actively corrupts data, even if it can be naively compiled and applied. The ease and practicality of making a live patch depend on when the code being modified is executed, if and how data structures are modified, and how those data structures are used. Some security updates also rely on updating processor microcode. Hooks can be used to check the safety of applying a patch or to edit existing data structures.

Live patch tools, at least for Xen, work by comparing the object code for a function pre and post patch. Using the same compiler and compiler options when making a livepatch as with the original executable is safest. There can also be changes to a function’s object code unrelated to changing its source code. Comparing the pre and post assembly is a useful tool for reviewing these changes and also as a cross-check for the safety and correctness of the intended update. Some specific examples and mitigations will be discussed.

Sometimes bugs in live patching code mean a live patch works on one processor family but not another. Live patches can also be stacked on top of each other. Live patches may exceed some payload size, or they may modify functions already modified by an earlier live patch. Care must be taken to avoid race conditions or invalid state if a logical patch cannot be applied as part of a single operation.

Some familiarity with C and what assembly and machine code is will be helpful for understanding this presentation. Live patching in the Xen hypervisor, which is simpler software than Linux, will be the primary example though Linux will be discussed at a high level.

This presentation is from the perspective of someone who generated live patches as one task among many, who sometimes opted not to live patch a change even if a live patch was hypothetically possible, and will not exhaustively cover all failure cases or methodology. Any use of information from this presentation is at your own risk. Presenter will not be responsible for any crashes, corrupted data, or devices set on fire from applying custom-built live patches.

Room:

Los Angeles AB

Time:

Saturday, July 30, 2022–18:00 to 19:00

Docs-as-Code for Open-Source Medical Hardware & PPE's

The advent of the COVID-19 pandemic created a global demand for accessible, reliable, & affordable medical devices &…

www.socallinuxexpo.org

Docs-as-Code for Open-Source Medical Hardware & PPE’s

Sean Marquez

Audience:

Topic:

Open Medical

The advent of the COVID-19 pandemic created a global demand for accessible, reliable, & affordable medical devices & personal protective equipment (PPE). However, as the pandemic progressed throughout 2020 and into 2021, the inventory of commercially available medical devices and PPE was severely restricted across almost all medical establishments throughout the world. This supply chain failure demonstrates the need to produce reliable and cost effective medical equipment and PPE without extensive dependencies on remote manufacturers. Volunteer groups formed in order to create PPE for medical workers, and others began to design PPE with a focus on local manufacturing and open designs. open-source organizations formed in order to fascilitate global volunteer efforts and contributions for developing medical devices and PPE for medical workers. The tight regulatory environment dictates standards for the design, manufacturing, & certification of medical devices and PPE’s. Such regulations commonly require a complete set of bill of materials, change management, and quality assurance processes. Meeting such regulatory requirements, while adhering to the principles of open-source hardware, poses its own set of technical challenges. This talk tells the story of how Tetra Bio Distributed (TBD), a volunteer-founded 501(c)(3) non-profit organization, developed medical devices and PPE as open-source hardware, using a docs-as-code approach.

Room:

Los Angeles C

Time:

Saturday, July 30, 2022–13:30 to 14:30

Snitching on Apps That Snitch On You

Kyle Rankin

Audience:

Topic:

Apps have a bad habit of snitching on their owners. While this is a much bigger problem on Android and iOS, it’s still valuable to detect when apps phone home on Linux. Unfortunately most Linux firewall software is designed for servers and focused on inbound connections.

This talk will cover using and tuning OpenSnitch, Linux software that prompts you whenever an app makes outbound connections and allows you to make sophisticated per-app rules to have tight control over your apps’ network connections. This adaptive software works both in desktop and mobile Linux and this talk will explore how I personally use OpenSnitch on my mobile Linux device.

Room:

Bel Air

Time:

Saturday, July 30, 2022–13:30 to 14:30

SUNDAY, JULY 31ST

Let’s open source diversity, equity, and inclusion!

Demetris Cheatham

Audience:

Atom Ramirez, Marie Ramirez

Diversity and inclusion within open source is being approached the way software development was prior to the creation of open source. Most efforts, in spite of significant resources and financial investment, are very fragmented, not openly shared, only address part of the challenges, and are not collaborative with those outside of a single organization or very few partnerships. We have to approach this differently in order to yield different results. We must drive diversity and inclusion as an open source community, utilizing relevant data and building on existing research and initiatives.

Room:

La Jolla

Time:

Sunday, July 31, 2022–10:00 to 11:00

Engaging Students with Customized Linux Images for Cybersecurity Training

Audience:

Topic:

Training Middle School and High Schools students in cyber security can be a challenging task especially with engaging students and keeping them on task. Come and learn how customizing Linux installations can help improve participation and engage students in solving cyber defense tasks. We will cover general topics of cyber defense, preparing a themed customized desktops, and using the Python programming to automate the customizations. The presenters, a sister and brother high school duo will share their experience with preparing themed Linux VMs for Cybersecurity training based from years of learning Cybersecurity through Cyberpatriots and other CTF competitions.

Room:

Century CD

Time:

Sunday, July 31, 2022–11:30 to 12:30

Using Cryptographic Hardware to Secure Applications

Bruce Momjian

Audience:

Advanced

Topic:

Using Cryptographic Hardware to Secure Applications

Cryptographic hardware allows storage of keys which cannot be easily viewed or copied. This presentation explains how cryptographic hardware can be seamlessly accessed by applications including openssh, gpg, and Postgres.

Presentation Link:

Room:

Century CD

Time:

Sunday, July 31, 2022–13:30 to 14:30

Network Visibility: The Heart of Modern Monitoring

Jeff Gehlbach

Audience:

Topic:

Observability

Regardless of the operational parameters and deployment models of the apps and services that power your organization, all modern applications depend on the network to deliver. The NetFlow family of protocols and their cloud equivalents play an important role in understanding how your workloads interact with the network, and how those interactions impact user experience. In this session we will dive into what flows are, how to interpret them to gain visibility, and how some real organizations have benefited from these insights by leveraging an entirely open-source monitoring stack.

Room:

Marina

Time:

Sunday, July 31, 2022–11:30 to 12:30

Expanding Open Source in Africa with Jenkins — Experience Report

Mark Waite, Alyssa Tong

Audience:

Topic: