Fixit 2: Meta’s next-generation auto-fixing linter

• Fixit is useless! Lengthy dwell Fixit 2 – the most recent model of our open-source auto-fixing linter.
• Fixit 2 permits builders to effectively construct customized lint guidelines and carry out auto-fixes for his or her codebases.
• Fixit 2 is out there right this moment on PyPI.

Python is among the hottest languages in use at Meta. Meta’s manufacturing engineers (PEs) are specialised software program engineers (SWEs) who give attention to reliability, effectivity, and scalability. They work on numerous tasks, together with debugging manufacturing companies, rewriting inefficient libraries, orchestrating venture deployments at scale, or capability planning and scheduling. And Python is usually one of many first instruments that PEs attain for, because it gives fast growth, simple to learn syntax, and a large array of open supply libraries.

Meta’s Python Language Basis staff — a hybrid staff of each PEs and conventional SWEs — helps personal and keep the infrastructure and tooling for Python at Meta. The staff helps engineers, information scientists, researchers, and anybody else at Meta utilizing Python to get their work finished.

One of many methods we accomplish that is constructing instruments that allow Python builders to put in writing higher, and extra dependable code extra effectively. This consists of instruments like automatic formatting and import sorting that remove tedium, or linters that information engineers towards maintainable code with fewer bugs.

This yr, we’ve got been constructing a brand new linter, Fixit 2, designed from the bottom as much as make builders extra environment friendly and succesful, each in open supply tasks and the various panorama of our inner monorepo. At Meta, we’re utilizing Fixit 2 with a number of early adopters, and plan to roll it out to the remainder of our monorepo quickly. However any developer can use it to carry out auto-fixing extra effectively and make quicker enhancements to their very own codebases.

Why a brand new linter? (why not X?)

There are a selection of fantastic linters within the Python ecosystem, a lot of which have a big neighborhood of third-party plugins offering a various array of lint guidelines. We have now used Flake8 internally at Meta since 2016, and it has been very profitable in serving to builders cut back bugs and hold a clear codebase. The favored flake8-bugbear plugin was even created by Łukasz Langa (writer of Black, PSF developer-in-residence, and launch supervisor for Python 3.8 and three.9) whereas working at Meta (then Fb), as a house for extra opinionated lint guidelines that we may each use internally and share with the remainder of the Python developer neighborhood. 

We even have a lot of inner plugins constructed by numerous groups, and Flake8 permits them to put in writing and allow customized lint guidelines immediately within the codebase with out getting sign-off from a central gatekeeper, and with out ready for a brand new deployment of Flake8 to roll out.

However whereas Flake8 has lengthy been a cornerstone of our linting resolution, it additionally has some tough edges. Writing new lint guidelines requires constructing total plugins (every claiming a portion of the “namespace” for error codes) and encourages builders to construct difficult plugins protecting a number of courses of errors. When these lint errors are discovered, Flake8 can solely level to a line and column quantity the place it occurred, however has no manner of suggesting modifications to the developer a listing of lint outcomes, leaving them in a state of trial and error to search out modifications that make the linter pleased. Additionally, Flake8 makes use of the stdlib ast module, making it unable to parse future syntax options and forcing builders to attend for instruments to improve earlier than they’ll use the shiny new hotness.

There are options to Flake8 after all, however a lot of them endure from a number of drawbacks: 

• A scarcity of assist for “native” in-repo plugins or customized lint guidelines. 
• Restricted or no assist for hierarchical configuration for various tasks inside a monorepo.
• No choice for auto-fixes when errors are discovered.
• Sluggish efficiency on giant codebases.

Whereas a few of these options aren’t essential, an important for developer effectivity is providing auto-fixes – automated instructed modifications that will fulfill the lint rule. This takes the guesswork out of utilizing a linter, and permits customers to rapidly assessment and settle for these modifications when doable, eliminating the necessity to re-run the linter till the code is lastly clear. Combining these auto-fixes with in-repo, customized lint guidelines gives a stage of tailor-made code high quality enhancements that’s laborious to beat.

Sadly, even Fixit, the auto-fixing linter that we constructed for Instagram and open sourced, didn’t assist native lint guidelines or hierarchical configuration – core necessities for our monorepo that’s house to hundreds of tasks, a lot of that are themselves open supply tasks with their very own distinct wants for linting and CI. We acquired many requests from builders to assist Fixit in our monorepo, however there have been sufficient hurdles that we have been solely capable of assist a small set of safety lint guidelines, decreasing the direct advantages to our Python codebase. 

Meet Fixit 2

After discussions with different groups, particularly within the quickly rising AI/ML area, we thought of our choices and determined upon a partial rewrite of Fixit. We deliberately designed the brand new model with an open source-first mindset, whereas incorporating the wants and necessities of our personal monorepos and open supply tasks from day one.

The framework and linting engine can be rebuilt from the bottom up whereas leaving the core design of lint guidelines largely untouched. The brand new system gives a hierarchical configuration primarily based on the TOML format; assist for native, in-repo lint guidelines just like Flake8; and a a lot improved CLI and API for integration with different instruments and automation. 

Fixit itself builds on high of one other Instagram open supply venture, LibCST, a concrete syntax tree for Python with a tree and node API following the patterns of the ast module in the usual library. The “concrete” a part of CST signifies that LibCST consists of each a part of the supply file within the ensuing tree after parsing, together with whitespace, feedback, and formatting components which are ignored by the ast module. That is what permits Fixit (and different instruments we constructed, like µsort) to securely modify supply recordsdata, with out utilizing common expressions or the chance of manufacturing damaged syntax, and gives the muse for Fixit to supply auto-fixes instructed by the lint guidelines themselves.

Writing a brand new lint rule will be finished with lower than a dozen strains of code, and take a look at circumstances are outlined inline. You possibly can even place it proper subsequent to the code that it is going to be linting:

# teambread/guidelines/hollywood.py
import fixit
import libcst
class HollywoodName(fixit.LintRule):
    VALID = [...] # no lint errors right here
    INVALID = [...] # dangerous code samples right here
    def visit_SimpleString(self, node: libcst.SimpleString):
        if node.worth in "'Paul'" or '"Paul"':
            self.report(node, "It is underbaked!")

Suggesting auto-fixes for the consumer is as simple as together with a brand new CST node when reporting an error:

def visit_SimpleString(self, node: libcst.SimpleString):
    if node.worth in "'Paul'" or '"Paul"':
        new_node = libcst.SimpleString('"Mary"')
        self.report(node, substitute=new_node)

Enabling this new rule inside the venture’s codebase will be finished with a easy config change:

# teambread/sourdough/fixit.toml
[tool.fixit]
allow = [".rules.hollywood"]

Now we are able to run our linter towards our venture:

# teambread/sourdough/baker.py
identify = "Paul"
print(f"howdy identify!")

$ fixit lint --diff sourdough/baker.py
sourdough/baker.py@7:11 HollywoodName: It is underbaked! (has autofix)
--- a/baker.py
+++ b/baker.py
@@ -6,3 +6,3 @@
def principal():
-    identify = "Paul"
+    identify = "Mary"
    print(f"howdy identify")
🛠️  1 file checked, 1 file with errors, 1 auto-fix out there 🛠️
[1]

The `lint` command solely reveals errors and instructed modifications. If we use the `repair` command, we are able to apply these instructed modifications again to the codebase:

$ fixit repair --automatic sourdough/baker.py
sourdough/baker.py@7:11 HollywoodName: It is underbaked! (has autofix)
🛠️  1 file checked, 1 file with errors, 1 auto-fix out there, 1 repair utilized 🛠️

Now that our auto-fixes have been utilized, we are able to verify that the venture is now clear and lint-free:

$ fixit lint sourdough/baker.py
🧼 1 file clear 🧼

When operating Fixit 2 with auto-fixing lint guidelines, any code that triggers the lint rule is a chance to get an automated substitute, enhancing the codebase with much less effort from the developer. Utilized extra broadly, Fixit 2 may even be used as a software to enact sweeping codemods towards a big codebase, whereas leaving a lint rule in place to deal with any matching code sooner or later.

Attempt Fixit 2 

Fixit 2 is on the market right this moment on PyPI. You possibly can set up and take a look at Fixit 2 with pip set up fixit. 

We have now a roadmap with plans for future enhancements and options, and a wealthy set of documentation and user guides that will help you get began with Fixit 2 in your individual tasks or repositories. We hope it proves helpful in your tasks, and we sit up for hearing your feedback!