Taming Your Go Dependencies

Internally at DigitalOcean, we had an issue brewing in our Go code bases.

Separate projects were developed in separate Git repositories, and in order to minimize the fallout from upgraded dependencies, we mirrored all dependencies locally in individual Git repositories. These projects relied on various versions of packages, and the problem was that there was no deterministic way to distinguish which project required what and when.

As a team, we knew this approach was not optimal, but coming to a consensus on a single way to manage packages was a tough decision. With a little bit of effort, we arrived at a solution which addressed the issue of managing package versions without needing an external management tool. We call our effort cthulhu, which is our Go repository. We also refer to it as a mono repo.

What’s a Mono Repo?

Building a cloud is fast-paced business. We have Go projects that serve APIs, move bits around from server to server, and crunch numbers. Because many of these projects share a common set of components, we determined it would be easier to create a single Git project and import all the existing projects. Here’s the high level structure of the project:

    .
    ├── README.md
    ├── docode
    │   └── src
    └── third_party
        └── src

It is a called a mono repo because we only have one repository. Our setup is straightforward. We have a root directory that serves as the base for cthulhu. Underneath this root, we have two additional directories: docode for our code, and third_party for other people’s code.

To develop Go software,set your GOPATH to $ {CTHULHU}/third_party/src:$ {CTHULHU}/docode/src. That’s it!
The reason that the third_party directory is listed first is to ensure that, when packages are fetched using go get, they’ll be installed in this directory’s src/ rather than docode.

At this point, you can create a script that can be sourced into a shell, and you can start developing.

Why Is This Good?

First and foremost, we believe the mono repo is a good idea because using it is frictionless. There are no arcane actions or sacrifices required to configure an individual developer’s workstation.

It is also beneficial because at this point of DigitalOcean’s Engineering team’s evolution, having a single repository for editing software means it is less likely for projects to get lost. Finding code is easy using the mono repo and our team’s simple conventions for naming services. We have three types of code: doge, our internal standard library, which contains code that is reused throughout the repository; services, which contains all of our business logic; and tools, which are one off applications and utilities used to manage our Go code, like our custom import rewriter that sorts and separates imports based on our current code guidelines.

    .
    ├── docode
    │   └── src
    │       ├── doge
    │       ├── services
    │       └── tools
    └── third_party

Because all of our Go is in a single repository, everything uses the same versions of external and internal dependencies. If a package is upgraded, every service which depends on the package receives the new functionality. This helps when dealing with security issues. It’s also nice to not have to manage versions explicitly. For our purposes, the canonical version is what’s under third_party/src. If your work requires an upgrade, you install the new dependency, run the tests, and then send a pull request.

It Isn’t All Rainbows.

Our mono repo is a great solution for us, but it doesn’t come without its own set of caveats.

One of the largest issues is actually an issue with Git. Git prescribes sub-modules for including dependencies in your main repository. When the sub-modules work correctly, there are no problems, but when they don’t work, it’s a thorny pain for everyone involved. In this case, we chose to sidestep the problem. Instead of dealing with sub-modules or an external management solution, we rename the git config directory (if there is one) for our dependencies. Because the .git directory doesn’t exist, Git considers the configuration to be just another set of files. If you want to upgrade the package, just revert the git directory name, and update. This isn’t an amazing experience, but it is simple.

Additionally, when you share a repository with all the other projects, you inherit all the other project’s issues. This means that if one of our individual services has a slow test suite, all services have a slow test suite. In general, testing Go is very fast. When you involve external tests, like database integration, things can slow down. A solution for this is to use the short flag to skip the long tests. An additional solution is to run tests for individual packages. The DigitalOcean Engineering team is still testing and deciding which solutions works best for us.

Where Do We Go Next?

Currently, our mono repo serves our needs well. It is an easy concept for newer developers to grasp, it doesn’t require any external dependencies, and it allows us to co-locate all of our Go code. In a nutshell, it’s a great thing for us and we believe it could be a great thing for other teams working with Go as well.

by Bryan Liles

DigitalOcean Blog

By: zedomax

yeah VPS.net used to be great 100% uptime for like a year straight for me then suddenly beginning of this year they went downhill, I’ve had about 12-24 hour downtime during their SAN update.  Now it’s like they do SAN update every week, time to get out, something happened with VPS.net, it’s like you said, 90% uptime at best or 75% for me.  This sucks since they don’t address the issue and fail to provide a solution.  

Comments on: VPS.net is Dead. Long Live ZippyKid.com!

Presenting FreeBSD! How We Made It Happen.

We’re happy to announce that FreeBSD is now available for use on DigitalOcean!

FreeBSD will be the first non-Linux distribution available for use on our platform. It’s been widely requested because of its reputation of being a stable and performant OS. While similar to other open source unix-like operating systems, it’s unique in that the development of both its kernel and user space utilities are managed by the same core team, ensuring consistent development standards across the project. FreeBSD also offers a simple, yet powerful package management system that allows you to compile and install third-party software for your system with ease.

One particularly compelling attribute of the FreeBSD project is the quality of their documentation, including the FreeBSD Handbook which provides a comprehensive and thoughtful overview of the operating system. We at DigitalOcean love effective and concise technical writing, and so we’ve also produced numerous FreeBSD tutorials to aid new users with Getting Started with FreeBSD.

We understand that this has been a long standing user request, and we’ve heard you. You might be asking yourself – what took so long?

The internal structure of DigitalOcean’s engineering team has rapidly changed over time due to the dynamic growth of the company. What began as a couple of guys coding furiously in a room in Brooklyn has ballooned to a 100+ person organization serving hundreds of thousands of users around the globe. As we’ve grown, by necessity we’ve needed to adjust and reorganize ourselves and our systems to be able to better serve our users. There have been many experiments in how we approach, prioritize and execute this work; this image is a result of the successful alignment of a few key elements.

Technical Foundation

Last year, we built our metadata service — allowing a droplet to have access to information about itself at the time that it’s being created. This is a powerful thing because it gives a vanilla image a mechanism to configure itself independently. This service was a big part what allowed us to offer CoreOS, and in building it, it gave us more flexibility in what we could offer moving forward. Our backend code would no longer need to know the contents of the image to be able to serve it. On creation, the droplet itself could query for configurables — hostnames, ssh keys, and the like — and configure itself instead of relying on a third party.

This fundamental decoupling is an echo of a familiar refrain: build well defined interfaces and don’t let knowledge leak across those boundaries unnecessarily. It’s allowed us to free images from customization by our backend code, and entirely sidestep the problematic issue of modifying a UFS filesystem from a Linux host.

Since we now had a feasible mechanism to allow images to be instantiated independently of our backend, we just needed to put the parts together that would allow us to inject the configuration upon creation. FreeBSD doesn’t itself offer cloud versions of the OS similar to what Canonical and Red Hat provide, so we started from a publicly available port of cloud-init meant to allow FreeBSD to run on OpenStack.

In order to query metadata, we need to have an initial network configuration in order to build our configuration, since DigitalOcean’s droplets use static networking. During boot time, we bring up the droplet on a v4 link-local address in order to do the initial query to the service. From there, we pick up the real network config, hostname, and ssh keys. The cloud-init project then writes a configuration that’s associated with the droplet’s ID. Linking this configuration to the droplet ID is the mechanism that allows it to know whether the image is being created from a snapshot or new create, or is just a rebooted instance of an already configured droplet.

Once this configuration has been injected, FreeBSD’s boot process can continue and use it accordingly — eventually booting into the instance as expected.

Focus

This endeavor began life as an experiment in how we organize ourselves in the engineering team. We were given a few weeks to pick a project, self organize in cross-functional teams, and execute. A lot went right during this process that allowed this project to succeed.

Deadlines are powerful things. Not in a punitive or negative sense of the word, but in a sense that there will be a well defined time where work on this will collectively end. So is having a very clear picture of what “done” looks like. In the case of BSD, it was particularly powerful to have a clear goal of a alpha functional BSD droplet with a date to drive for. Given the freedom to focus on a single goal, clear communication, and well defined restraints, we were able to finally deliver a long standing user request with relative ease.

This is the start to the many things we’re excited to build in 2015!

By: Neal Shrader

DigitalOcean Blog

5 Basic Benefits of Google Business Apps

business-app-userWith apps available everywhere, sometimes it’s hard to decide what apps you should use for your business. When choosing an app it’s important to know how an app will benefit your business.  If you’ve spent any time on the Internet, you already know the name Google, so it shouldn’t surprise you that Google has some of the best business apps available.  Here are five benefits of using Google Business Apps.

Easy to Start and Use

Unlike many apps, the Google Business Apps are intuitive.  Calendars, email, file sharing, contacts, and more are simple to use and will work virtually on any internet connected device.  Google Business Apps automatically update when there is a new feature or upgrade, you no longer have to wait for a new feature, unless your system administrator wants to manually roll out features as required.

As Powerful As You Need It

Google Business Apps has the Google apps script and Google for Work APIs enabling your system’s administrators and software developers to integrate their software with them.  Along with third party integration you will find endless possibilities and extensions enabling you to use Google for Work. 

Cost Efficient

For about a cup of latte from your favorite coffee bistro a month, Google Business Apps can give you the tools to expanding your business.  For only $ 5 a month, you will receive email addresses for your team with your company’s name, 30 GB storage you can use for file storage and sharing, online calendars, and the ability to easily create online spreadsheets, slides, text documents, and more. All these great features including admin controls and security from a name you can trust. If you prepay for a year you will actually save $ 10. 

Security

The safety of your data is top priority at Google. The company is FISMA-Moderate level certified — this is the same level of certification for the internal email usage within the United State’s government. Google is also capable of supporting HIPAA compliance. Google is trusted by millions to virtually secure their email from any threats through routinely checking emails before downloading a document for any threats of viruses, pshing emails, malware and more.

Mainstream Support

Google Business Apps are intuitive, however there may come a time when you need assistance with a certain app. There are a range of resources with excellent support to answer your questions and concerns. You can choose to seek help from Google Certified Admins support, you can also seek advice through Google online forums, online support websites such as Google Gooru,  and companies specializing in Google support. With an array of Google support outlets you can quickly find a solution.

This is why Google Business Apps are the right choice for your business and why you should consider it.  Google Business Apps has the security, reliability, support, and the cost efficient features that will help place your company at an advantage above the rest. Discover how you can lead your company through the use of Google Business Apps today.


Back to Business

By: Socrates

Perhaps it is time for you to move away from VPS.net too my friend?!

Comments on: VPS.net is Dead. Long Live ZippyKid.com!

By: Simon Dann

I have had a couple of really alarming issues with vps.net aside from the up and down nature of their service I run a debian 6.0 VPS with them which I cant update because each time I do it breaks due to a flaw in the container that they know about but wont fix.

I also run a number of servers from home for svn, email and personal blog and have managed to keep a 99.9% up-time record for the past six months, better than what my records show I am getting from vps.net

Initially they were really good, customer support was top notch and I couldn’t fault them, but as the year has progressed their support’s English skills have diminished and the outages continue to rise both in number and length.

Maybe time to move myself.

Comments on: VPS.net is Dead. Long Live ZippyKid.com!

What’s Your Libscore?

The contributors to Libscore, including our own Creative Director Jesse Chase, wanted to offer this post as a thank you for all the support the project has received. Julian Shapiro launched Libscore last month hoping that the developer community would find the tool useful, and continues to be grateful for all of the positivity and constructive feedback throughout the web.

For those who haven’t heard, Libscore is a brand new open-source project that scans the top million websites to determine which third-party JavaScript libraries they are using. The tool aims to help front-end open source developers measure their impact – you can read all about it here.

In this post, we’ll break down the technology that Libscore leverages and discuss some of the challenges getting it off the ground. We were also lucky enough to talk with Julian and get some insight as to where he sees the project going.

Thomas Davis: A Technical Overview

Unlike traditional web crawlers, Libscore thoroughly scans the run-time environment of each website into a headless browser. This allows Libscore to monitor the operating environment on each website and to detect as many libraries as possible – even those that have been pre-bundled and required as modules. The tradeoff, of course, is that running one million headless browser connections is much more resource intensive than performing basic cURL requests and parsing static HTML.

The biggest insight we gained while designing the crawler is that the best way to weed out false positives for third-party plugins is to leverage the broader data set we’re aggregating. Specifically, we weed out third-party libraries that didn’t exist on at least 30 sites out of the 1 million crawled. Using meta-heuristics like these allowed us to more confidently detect libraries that were in fact third-party plugins, and not just arbitrary JavaScript variables that were leaking to the global scope.

On the backend, crawls are queued via Redis with the results stored in MongoDB. Both services are loaded fully into RAM which allows our RESTful API to serve requests faster than it would querying the disk. The main bottleneck to crawling concurrency is network bandwidth, but thanks to DigitalOcean, it was a breeze to repeatedly clone instances and run crawls during off-peak times in different regions. Ultimately, using just a few high-RAM DigitalOcean instances, we parse 600 websites per minute and complete the entire crawl in under 36 hours at the end of each month.

As the crawler runs, raw library usage data for each site is appended to a master JSON file, which we simply read from the file system with Nodejs. Once all the raw usage data is collected we start a process dubbed “ingestion”, which is responsible for aggregating the results and making them accessible via the API. We actually attempted to load the entire dataset into ram to perform our calculations, but quickly ran into a quirky problem with V8 not being able to allocate anymore than approximately 1GB of memory for arrays. For now, we are splitting up the raw dump into smaller files to bypass the memory limit, though in the future we might just rewrite the project to use a more suitable language and environment.

Jesse Chase: Design Improvements

While Libscore currently serves as an invaluable tool for surfacing library adoption data, the future is even more exciting. To illustrate it let’s jump ahead 6 months – smack in the middle of summer. At this point, Libscore will have crawled through the top million sites 6 times already (or 6 million domain crawls!), bringing forth rich month-over-month trend data on library usage.

By providing users with a soon-to-be-released time series graph, with the ability to plot multiple libraries over the same time period, developers will gain new insights into how libraries are changing over time. For example, users will be able to see why a library’s usage plummeted from one month to the next – potentially due to the increased adoption of another library. Soon, this data will be fully visualized.

Julian Shapiro: The Future Of Libscore

Libscore is more than a destination for JavaScript statistics; it’s also a data store that can be leveraged in the marketing of open source projects. One way we’re enabling this is via embeddable badges that showcase real-time site counts. Open source developers can show off these badges in their GitHub README’s, and journalists writing about open source can similarly include them to provide context on the real-world usage of libraries.

In addition to badges, we’re also releasing quarterly reports on the state of JavaScript library usage. These reports will showcase trends, helping developers learn which libraries are rising in popularity and which are falling. We hope these reports will become a valuable contribution to discussions around the state of web development tooling, and to finally provide the community with concrete data they can use to make decisions.

Creator and developer – Julian Shapiro
Backend developer – Thomas Davis
Creative Director – Jesse Chase

DigitalOcean Blog

By: Socrates

I am sorry to hear about your problems Simon but I am not surprised. By now I have heard a few horror stories from VPS.net customers. All I can say is this: I have been dreading the move away from VPS for a while but once I did it, I have never looked back. Perhaps, it is time for you too?!

Comments on: VPS.net is Dead. Long Live ZippyKid.com!

By: Nirajan Bom Malla

same with my, i moved to vps 5 months ago, i see several server fail, in feb 2 atlanta location disk failed and i lost all my new datas, which they failed to recover. Choosing VPS.net is really a great mistake, they claim and advertise with 100% update, no server have that, 99.9% is acceptable. Also they dont have Phone Support. 

I mentioned in my blog http://blog.detechter.com/very-bad-experience-with-vps-net-cloud-hosting/

 

Comments on: VPS.net is Dead. Long Live ZippyKid.com!

By: Anonymous

Are you still using Zippykid.com on this website up to today?

Reason I ask is vps.net has once again gone down and I am done with giving it yet another second chance.

Comments on: VPS.net is Dead. Long Live ZippyKid.com!

Linux Web Hosting Package

Web designers often look for the web hosting packages and services before executing the plan of their web designing. There is wide variety of web hosts services available which compiles all the needs of the web designers and offer excellent services. But to choose from the plethora of services is a tedious task. Before executing their plan of designing website, the first step is to choose the appropriate domain name that suits the website. Web hosting package is the next and most important step where you will need to upload your files for configuration.

Web hosting packages plays an important role in the web designing. Various web hosting services are currently available in different operating systems such as Linux hosting, VPS hosting , Windows etc. There are certain essential ingredients which are needed for every hosting plan. It is important for web designers and webmasters to choose the appropriate and best suited web hosting plans and services.

One of the latest and common hosting plans is Linux hosting, which is more secure and efficient than other operating systems as well as hosting plans available in today’s era of advance technology. It is indeed considered as the most popular hosting services since it is freely available software around the web. This plan comes with a bunch of services and components such as Apache server, PHP language and mySQL database system. High performance, greater stability and even lower cost are the key features of the Linux operating system and are preferred by most of the designers due to its huge advantages over other operating systems.

Advantages of Linux Hosting web services:

There are various advantages that make this service stand out in today’s era and make it a preferable choice among the designers and webmasters.

  1. 1.Availability: This web hosting services are freely available around the web. Since it is open source software, designers don’t have to invest higher budget in hosting and the software can be changed by any person according to the requirement.
  2. 2.Secure and Stable: The key features of this hosting service are its security and stability which yield high productivity and efficient results in terms of performance.
  3. 3.Low Cost: The web hosting service suits best for those individuals who do not have high budget, the web hosting plan is quite cheaper and can be the preferred solution.
  4. 4.Combined Package: This hosting plan comes with a bundle of services comprising of FTP Access, Common gateway interface (CGI) mechanism, mySQL Database systems. These services make this hosting plan different from rest of others. This is the main reason that most designers find it an appropriate web hosting plan in today’s time.
  5. 5.Effective Administration services: The administration services offered by this platform are very effective and provide you excellent results.
  6. 6.Reliable: The above hosting plan is very reliable as it provides you with a facility of solving every technical problem that can arise during the working. Its reliability, security and performance are added features and advantages of this hosting plan.