Gophercon 2023 : A summary of my learnings

In this blog, I'll be sharing my insights on some of the engaging discussions that took place at GopherCon India 2023. These talks expanded my horizons, and I hope you find this blog useful

Talk 1: Rubiks-Kube

The speaker discussed solving a Rubik's cube using sensors and GoLang. They used a Lego robotics kit to demonstrate. and talked about the history of Raspberry Pi and its integration with Lego robotics kits.

• Go supports ev3dev, a Debian-based OS for Lego Mindstorms.

• SCP was used to transfer the Go program to the BrickPi controller. This required setting GOOS, GOARCH and GOARM for cross-compatibility.

• The source Go code is compiled into a binary that runs on BrickPi. This involves several steps:

Go format -> assembly -> object file -> binary.

BrickPi uses a statically linked binary.

• There were 3 main steps to solving the cube: detecting the face colors, finding a solution, and performing the moves.

• A color sensor was used to detect the face colors and convert them to HSV values.

• The "master's formula" algorithm was used to find a 64-step solution.

• Motors and a flipper were used to perform the moves.

• The code had 3 implementations for different microprocessors, selected using a switch.

• The makefile specified the GOOS, GOARCH and GOARM flags for compiling, and used scp to transfer the binary.

That covers the main highlights from the talk.

Talk 2: Is unit testing an afterthought or a necessity?

Why unit test:

• Unit testing helps find bugs early before they impact other code. It reduces integration issues and improves code quality.

• Unit tests have the highest return on investment compared to integration and functional tests since they are focused on isolated units of code.

Writing good unit tests:

• Write tests for different test cases of each sub-block within a function. Test edge cases.

• Write testable code by ensuring functions have a single responsibility and minimal dependencies.

But as for testing complex functions:

• We can mock external dependencies like HTTP calls by directly accessing the methods for testing. This isolates the unit being tested.

• Use table-driven tests to auto-generate test cases. Tools like VSCode can help generate test fixtures.

Pros and cons of unit testing

Pros:

• Finds bugs early
• Improves code quality and maintainability
• Saves time by preventing breaking changes

Cons:

• The initial cost of writing tests is the time consumed.

• Maintenance overhead as code changes.

What I Concluded:

• Unit testing should be an integral part of the development process, not an afterthought.

• The benefits of finding and fixing bugs early are increased confidence and maintainability which outweigh the initial cost of writing tests.

• Unit tests help evolve code by acting as documentation and guardrails for future changes.

• As applications grow larger, unit testing becomes even more critical to manage complexity and limit regression issues.

The ROI is highest when tests are written alongside the code being tested.

Talk 3: Generics in Go

Reflection vs Generics in Go:

Reflection allows operating on types at runtime but allows bypassing the type system. It can lead to bugs that only show up at runtime.

Generics allow defining type-parametrized functions and types at compile time. This catches errors at compile time and generates optimized code.

Type vs Kind:

Type refers to the concrete type of a value - int, string, etc.

Kind refers to the more abstract category that a type belongs to - integer, slice, struct, etc.

Every type has a kind, accessed via reflection's Kind() method.

Generics use:

• Type parameters like T int and U string to parametrize functions and types.

• Type sets to constrain what types are allowed as parameters via the interface{} blank interface.

• Type inference to deduce concrete types from usage.

Go implements generics via monomorphization (generating a separate compiled version for each concrete type used.)

This has pros like performance but cons like larger binaries.

When to use Generics:

• When you want to define reusable, typesafe abstractions that catch errors at compile time.

• For performance-sensitive code you want optimized code for each concrete type.

In summary, generics in Go represent a major step forward by allowing typesafe abstractions that match the performance and compiled nature of Go. They address many of reflection's shortcomings while still fitting Go's philosophy.

The talk highlighted how generics, although a simple addition, Add a wonderful layer of utility to Go's existing type system in a pragmatic way significantly increasing the productivity and maintainability of Go code.

Talk 4: Metaprogramming in Go

Go does not have native support for metaprogramming features like macros or a reflective meta-object protocol. However, there are still ways to achieve some metaprogramming in Go:

Reflection:

• The reflect package allows inspecting and manipulating Go values and types at runtime.

• This enables some metaprogramming use cases like serialization, ORM and deep equality checking.

• However, reflection comes with performance overhead and reduced readability. Runtime-type assertions can cause errors.

Code Generation:

• Generating Go code from templates or programmatically constructed instructions.

• The text/template package can be used to generate Go source files from templates.

• Code generation separates logic from boilerplate, making code more maintainable and consistent.

• Productivity is increased by focusing on the core logic and automating repetitive tasks.

Benefits of metaprogramming in Go:

• DRY (Don't Repeat Yourself) principle

• Increased abstraction

• Flexibility and extensibility

Drawbacks:

• Reduced readability

• Potential performance impact

• Runtime type errors

While Go does not natively support many metaprogramming features, reflection and code generation techniques allow us to achieve some metaprogramming capabilities. This gives Go some flexibility at the cost of reduced readability and performance in some cases. Developers must weigh the trade-offs for their use cases.

Talk 5: Fuzz testing

Fuzz testing is an automated software testing technique that involves feeding large amounts of random inputs to a program to discover crashes, security flaws, and logical errors.

How fuzz testing works:

• A fuzz engine or fuzzer generates random or mutated inputs and feeds them to the program.

• A seed corpus of known valid inputs is used to guide the fuzzer and generate new inputs that are similar but not identical.

• An internal or generated corpus of inputs is used to continuously fuzz the program.

Benefits of fuzz testing:

• Finds edge cases and bugs that human testers miss due to biases.

• Finds vulnerabilities that attackers could exploit.

• Identifies functional issues, crashes, unexpected behaviors, and invariants.

Limitations:

• Difficult to find logic bugs.

• High computational requirements.

• Tests one program at a time.

Tips for effective fuzzing:

• Make the fuzz target (program under test) fast and deterministic.

• Avoid global state and dependencies.

• Make the code testable.

So....fuzz testing is an automated technique that can find many critical bugs but has some limitations. Fuzz testing, when used alongside other testing techniques, can significantly improve software reliability and security.

Talk 6: Cache design patterns

When designing a cache in Go, there are some key considerations:

Thread safety: The cache must be able to handle concurrent access from multiple goroutines.

Locking: Appropriate locking mechanisms are needed to synchronize access to the shared cache data.

Scaling: The locking design should allow the cache to scale to handle a large number of requests.

Common locking patterns in Go include:

• Mutex: A basic mutual exclusion lock that ensures only one goroutine can access the critical section at a time. This makes the cache thread-safe.

• RWMutex: A read-write lock that allows concurrent reads but exclusive write access. This can improve performance for caches with many reads and infrequent writes.

• Optimistic locking: Avoiding locks until a conflict is detected. This reduces lock duration and the potential for contention.

• Granular locking: Applying locks at a finer level of granularity. This can improve parallelism and scale.

However, excessive locking can lead to issues like unstable CPU usage and lack of parallel progress.

Alternative lock-free designs use techniques like:

• Atomic operations: Functions like CompareAndSwap that perform reads and writes atomically.

• Atomic snapshots: Taking an atomic copy of shared data. Arrays combined with atomics and data structures like bloom filters can provide lock-free snapshots.

appropriate locking and unlocking designs are crucial for concurrent caches in Go. Both locking and lock-free techniques have trade-offs and the right approach depends on the cache use case and scaling requirements.

Talk 7: Checkpoint Restoration Using CRIU

CRIU (Checkpoint Restore In Userspace) is a tool that allows checkpointing and restoring of running Linux containers and processes.

Why C/R is useful:

• Software failures are common and taking backups of running processes can be difficult.

• Checkpoint and restore allows "freezing" a running process and restarting it from the same state later. This provides fault tolerance and facilitates migration.

How CRIU works:

• CRIU attaches to the process using ptrace and makes it a child process.

• It collects information about the process tree, open files, namespaces, memory mappings, etc.

• It injects "parasitic" code into the process to collect its resources.

• It serializes the collected data into binary files using Protocol Buffers for storage.

Challenges of checkpointing:

• Library version mismatches between checkpoint and restore.

• Maintaining consistency of PIDs and other IDs.

• Cannot checkpoint processes that have isolated parent processes.

In summary, CRIU enables checkpointing and restoring of running processes by serializing their state. This allows processes to be "frozen" and restarted from the same state.

However, checkpointing faces challenges due to the complexity of process state. CRIU aims to address these challenges, but limitations still exist.

Talk 8: Dive into debugging distributed systems using Delve

Dgraph:

Dgraph is an open-source, distributed graph database. It uses a graph model to store and query data.

Advantages of Dgraph's graph query language (DQL) over SQL:

• Supports graph traversals - SQL does not support traversing relationships between entities.

• Less complicated - DQL queries are simpler than SQL queries for complex graph operations.

• Flexible schema - Dgraph has a dynamic schema that can evolve over time.

Dgraph's architecture has two main components:

• Zero - The coordinator that manages the cluster and assigns tasks to Alpha nodes.

• Alpha - The worker nodes that store and query the graph data.

Delve:

Delve is a debugger for Go programs. It allows setting breakpoints, inspecting variables, and more.

Debugging with Delve:

• Reproduce the issue in a test.

• Identify the component - service, worker node, etc.

• Debug that component using Delve by

executing the following steps:

1. Compiling the test with debugging symbols.

2. Running Delve on the executable.

3. Setting breakpoints.

4. Stepping through code, inspecting variables, etc.

This helps debug distributed systems by focusing on a specific component in isolation.

In summary, Dgraph is a distributed graph database with benefits over SQL.

Delve is a useful tool for debugging Dgraph and other Go programs, allowing you to focus your debugging on specific components in isolation.

Talk 9: Hardening Go Concurrency: using Formal methods to verify correctness

TLA+:

A formal specification language which is used to describe systems and verify their correctness. TLA+ uses mathematical logic and set theory to specify systems rigorously and unambiguously. It allows:

• Describing a system using mathematical constructs like sets, functions and sequences.

• Modeling states and state transitions using state machines.

• Formally specifying properties the system must satisfy.

• Model checking to verify that the system meets its specifications.

Model checking: An automated technique for verifying that a system meets its specifications. A model checker exhaustively explores all possible states of a model to check if desired properties hold. If a property is violated, a counterexample trace is generated.

Liveness property: A type of temporal property that specifies something good will eventually happen. For example, "every request will eventually get a response". Liveness properties are important to check for deadlocks and livelocks.

Example: The speaker Gave us an example of a Concurrency Bug from the [google/gops] repository, where the issue was: running into a Deadlock state when no. of goroutines > 10. The issue was solved using the statewide approach for debugging.

The statewide approach to debugging involved:

• Modeling the system and its possible states in TLA+

• Formally specifying the expected behavior and liveness properties

• Using model checking to find a counterexample trace that revealed the deadlock state

• Fixing the bug based on the trace

Miscellaneous concepts I heard for the first time:

Sharding:

The process of splitting up large databases into smaller parts called shards. This improves scalability and performance by distributing the data across multiple servers. Sharding can be done horizontally by splitting tables, or vertically by splitting columns. It requires a sharding key to determine which shard a piece of data belongs to.

Runes:

The internal representation of Unicode characters in Go. A rune is an alias for int32 and has ability to represent any Unicode code point. Strings in Go are UTF-8 encoded, while runes represent single Unicode characters.

Locking:

The mechanism to control access to shared resources in a concurrent program. Locks ensure that only one goroutine can access a resource at a time. example: Go provides sync.Mutex for mutual exclusion locking.

Logs:

Crucial for debugging, and monitoring software systems. Go's log package provides flexible logging with configurable output, format and severity levels. Logs can be written to files, the console or networked endpoints.

Rate Limiting:

The technique of restricting the rate of requests to a resource to avoid overloading the system. Rate limiting can be implemented using -> leaky bucket algorithms, fixed window counters or tokens.

It is important for APIs, web applications and network services.

eBPF:

An in-kernel virtual machine that allows executing programs inside the Linux kernel. eBPF can be used for tracing, networking and security purposes. Go has good support for writing and loading eBPF programs.

Conclusion

GopherCon 2023 was an amazing experience. I met so many talented and passionate Gophers from all over India who are building incredible things with Go. Discussing code and ideas with like-minded people filled me with excitement and motivated me for what's possible when we come together as a community.

until next time

Happy Coding With Go!