Blog Posts | Democratizing Data

TikTok kept letting strangers create accounts with my email, so I filed PIPEDA and CASL complaints

Fri, 24 Apr 2026 18:53:00 -0700

Disclaimer

This is a personal account of how I, a non-lawyer Canadian resident, used PIPEDA and CASL to address a privacy issue with TikTok. It is not legal advice. Procedures, deadlines, and applicable laws change. If you are facing a similar situation, please verify everything against current OPC and CRTC guidance, and consider consulting a lawyer for matters with significant legal or financial stakes. Parts of the research and drafting in this article were done with the help of an LLM. I reviewed the output, but factual errors are still possible. Please check the primary sources I link to before acting on anything here.

TL;DR

TikTok accounts I never created kept showing up under my email address. The first one was @chezou2. After it was deleted, a different handle appeared.
TikTok lets people register accounts without verifying that they control the email address. This is a structural failure to obtain consent.
TikTok Support eventually deleted the first account after I escalated. A different account appeared shortly after the deadline I had given them.
I filed two complaints with Canadian regulators:
- A violation report to the
- A formal complaint to the
The OPC had already issued a joint Report of Findings against TikTok in September 2025 ( ), so my complaint joins an existing thread of regulatory attention.
Below is what I did. Whether anything from this is useful for your own situation is for you to judge.

What is PIPEDA and CASL?

If you live in Canada and you have never come across these acronyms, here is the short version.

PIPEDA stands for the Personal Information Protection and Electronic Documents Act. It is the federal law that governs how private-sector organizations collect, use, and disclose personal information in the course of commercial activity. Your email address is personal information under PIPEDA. Complaints about PIPEDA go to the OPC, the Office of the Privacy Commissioner of Canada.

CASL stands for Canada’s Anti-Spam Legislation. It regulates the sending of “commercial electronic messages” (think marketing emails) without consent. Complaints about CASL go to the CRTC through its Spam Reporting Centre.

The two laws cover different things. PIPEDA is about what an organization does with your personal information. CASL is about whether an organization is allowed to send you a particular email. In my case, both were relevant for different reasons.

What happened

In November 2025, I started receiving emails from noreply@account.tiktok.com reporting “a new device login.” The login was from an Android phone I do not own. More fundamentally, I have never created a TikTok account.

The emails were addressed to my private email and referenced a handle, @chezou2, that someone had registered with a name close to mine. There was an awkward fact behind this: TikTok lets people register accounts without verifying that the email address belongs to them. No “is this you?” confirmation email had ever arrived. The account was simply created, and I, the actual address holder, was on the receiving end of TikTok’s automated correspondence.

This is a structural problem, not an incidental one. Anyone can register a TikTok account using anyone else’s email address, and the actual address holder receives the platform’s notifications indefinitely.

March 2026: contacting TikTok directly

On March 15, 2026, I submitted a “Report a potential privacy violation” through TikTok’s privacy report form at .

The first reply, from feedback@tiktok.com, was a templated message: “If you want to delete your account, log in and follow these steps.” The reply assumed I owned the account. The whole problem was that I did not.

I replied, explaining that the account was not mine, that someone had created it using my email address without my consent, that I could not log in because I had never created the account, and that PIPEDA applied. The next reply asked me to verify ownership: signup date, first login location, registered device, registered phone number, linked third-party accounts.

I could not answer any of these, and that was precisely the point. I replied saying so. My inability to provide ownership details was itself evidence that I had not created the account, and a process that demanded ownership details from a non-user was structurally incompatible with non-user privacy complaints. I also stated that if the matter was not resolved within 30 days, I would file a formal complaint with the OPC.

After further escalation, @chezou2 was eventually deleted. I did not get an explicit confirmation; I just noticed, when I checked on April 16, that the profile no longer existed.

April 2026: the same problem, again

The same day I confirmed @chezou2 was gone (April 16, 2026, the day after the 30-day deadline I had given TikTok), I started receiving emails from a different sender (notification@service.tiktok.com, with Reply-To: edm.feedback@tiktok.com). This time addressed to a new handle I also did not own. Messages in French, purporting to be social notifications from various TikTok users.

After @chezou2 was deleted, someone else had created another account using my email address, and TikTok’s notification machinery had picked up where it left off.

This made the issue clear. Deleting individual accounts will not fix anything as long as TikTok does not verify email ownership at registration. The cycle can repeat indefinitely.

A small but telling detail: the Reply-To address on these emails is edm.feedback@tiktok.com. In email marketing, “EDM” stands for “Electronic Direct Mail.” TikTok’s own infrastructure treats these notifications as part of its marketing email stack. This matters under CASL, as I will get to.

Two regulatory routes in Canada

I decided that filing individual deletion requests was a losing game and turned to the regulators. There are two distinct complaints that apply.

CASL → CRTC

CASL regulates the sending of unsolicited “commercial electronic messages” (CEMs). The relevant points for my case:

The CRTC has the power to impose substantial monetary penalties for violations.
TikTok’s notification emails encourage engagement with the TikTok platform. They can plausibly be characterized as promoting commercial activity.
The transactional or account-notification exemption should not apply here, because I am not the account holder. TikTok has not verified, and structurally cannot verify, that my email address belongs to the account being notified.

The reporting channel is the at spam@fightspam.gc.ca. A useful detail: you can simply email reports there, and forwarding the offending messages as .eml attachments preserves the full headers. This is better evidence than what the online form captures.

I sent one context email and attached five sample messages in .eml format.

A caveat on expectations: the SRC does not provide individual case feedback. Submissions feed an enforcement-targeting dataset; you do not get a “your case has been resolved” notice. The CASL route is about contributing to enforcement signal, not about getting a personal resolution.

PIPEDA → OPC

PIPEDA governs the collection, use, and disclosure of personal information. The relevant provision here is Schedule 1, , read together with (Valid consent).

Formal complaints to the OPC are filed through their . Two things to know in advance.

The form’s automatic stop

The form asks whether you have filed a complaint about the same matter with another body. I had filed with the CRTC SRC eight days earlier, so I answered “Yes.” The form ended my session: “If your complaint to another body covers any of your concerns with your personal information, please end this session and complete that complaint process first.”

The problem: the SRC does not issue case outcomes. Waiting for that process to “complete” would mean never filing a PIPEDA complaint at all. The OPC’s online form was treating CASL and PIPEDA as overlapping, when in fact they cover different conduct.

Recovering through the Information Centre

The OPC’s accepts free-text inquiries up to 2,000 characters. I sent one explaining the situation, the difference between the CRTC and PIPEDA matters, and asked how to proceed past the form’s stop.

A reply came within a day. The OPC suggested:

Contacting TikTok’s privacy contact (TikTok Inc., Culver City) in writing first.
Then filing a formal complaint if unresolved.
For the form’s “complaint with another body” question, answer “No” and explain the situation in the free-text section.

The first point was easy. I had already done the equivalent in March, through TikTok’s own privacy report form (the same channel the OPC pointed me to) and the email correspondence that followed.

Filing the formal complaint

The OPC online form is structured in three parts. Part A asks about steps already taken. Part B determines jurisdiction. Part C covers details and remedy. Part C has four free-text fields with character limits ranging from 500 to 2,500.

The argument I built:

Principle 4.3 (Consent), read with s.6.1 (Valid consent). TikTok cannot have any basis to believe the address holder consented, because it has no mechanism to verify email ownership. Section 6.1 requires that “an individual to whom the organization’s activities are directed would understand the nature, purpose and consequences” of the collection, use, or disclosure. TikTok cannot satisfy this when it has not even confirmed who the address holder is.
No s.7 exception applies. I am not a customer; there is no investigation, emergency, or any other listed basis.
Reasonable expectations of a non-user. PIPEDA Report of Findings (Facebook using non-members’ email addresses to suggest friends) is directly on point. The Commissioner there found that a non-user could not reasonably expect a platform to use their email to create social connections, particularly when there is no prior relationship to the organization. The same logic applies to me and TikTok.
Recurrence as evidence. A second account appearing after the first one was deleted demonstrates that the issue is systemic, not an isolated incident.

What I asked for, in order of priority:

A Compliance Agreement under requiring TikTok to implement email verification at registration and a non-ownership-based privacy complaint process.
Deletion of all accounts currently associated with my email.
Adding my email to a registration blocklist.
A public Report of Findings, given the systemic nature of the issue.

Attachments (within the 8-page and 25 MB limit): the March TikTok Support thread as a PDF, the OPC Information Centre response, the November security notification, and one representative April notification message.

Existing OPC × TikTok context

Worth knowing: in September 2025, the OPC, the CAI (Quebec), the OIPC BC, and the OIPC AB jointly issued on TikTok. That investigation focused on minors’ consent and targeted advertising, not on non-user account creation, but the regulators have already concluded that TikTok’s overall consent practices have problems.

That report also addressed the BC PIPA and PIPEDA jurisdictional question. In cross-border data flows, the federal and provincial laws operate in an “airtight seal” rather than excluding each other. As a BC resident filing against a Singapore-based organization, this confirms PIPEDA jurisdiction is appropriate.

So my complaint is not an isolated grievance. It joins an existing thread of regulatory scrutiny, from a different angle.

What I did, in summary

If you found this post because you are dealing with a similar situation, here is the sequence of what I did. The disclaimer at the top of the post still applies: this is not legal advice, and your situation may differ in ways that matter.

Step 0: Preserve evidence.

Save the suspicious emails: sender, date, body, and full headers.
If they are in your spam folder, rescue them before they auto-delete (Gmail purges spam after 30 days).
Note the unauthorized account handle(s) referenced in the messages.

Step 1: Contact the organization in writing.

Use whatever privacy reporting channel the organization provides (form, email, etc.).
Cite PIPEDA explicitly. It changes the tone of the conversation.
Make it clear you are not the account holder and are demanding deletion as the email holder, not as the user.
Set a reasonable deadline. Mine was 30 days. This is not a statutory requirement, just a tactical one.

Step 2: Report to the CRTC SRC under CASL (if the messages have any commercial or promotional flavor).

Email spam@fightspam.gc.ca directly. Forward the offending messages as .eml attachments to preserve headers.
Do not expect individual case feedback. The SRC uses submissions for enforcement targeting.

Step 3: File a PIPEDA complaint with the OPC.

If unsure how to proceed, send an inquiry to the OPC first (2,000-character limit; replies typically come within a day or two).
File the once you have a clear path.
Attachments are limited to roughly 8 pages and 25 MB. Choose evidence carefully. The full thread of correspondence with the organization is the most important.

Things to be aware of.

The OPC has a current backlog. Investigations can take months.
The OPC does not have order-making power. It cannot impose fines or issue binding orders directly. The strongest tool it has is the Compliance Agreement under s.17.1, which is enforceable in Federal Court. Worth asking for explicitly.
BC OIPC may have concurrent jurisdiction, but in cross-border situations the OPC handles routing.

A note on AI-assisted research

Parts of the research, drafting, and form responses for this complaint were done with the help of an LLM (Claude). It was useful for surfacing precedents, building arguments structurally, and drafting boilerplate quickly. It was also wrong about some specifics. For example, an early version of the analysis presented “30 days for organization response” as a PIPEDA requirement, which it is not. I caught that on a later pass, but the principle is general.

If you do this kind of research with an AI assistant, treat its output as a starting point, not as a citation. Verify every legal reference (statute number, section, finding number, deadline, jurisdiction) against the primary source, such as laws-lois.justice.gc.ca for legislation, priv.gc.ca for OPC findings, or fightspam.gc.ca for CASL, before acting on it. AI tools hallucinate, and acting on a hallucinated regulatory deadline is the kind of mistake that has real consequences.

For me, the value was in the working pattern. The LLM did the research scaffolding, I verified the primary sources, and I made the decisions about what went into formal submissions. That division of labour was practical and, I think, the right way to use these tools for legal-adjacent work.

Important

Reminder: Not legal advice. Verify primary sources. Consult a lawyer if your situation involves significant legal or financial stakes.

Embedding workflow templates in skills: shifting the LLM's role from "generation" to "rendering"

Sat, 28 Mar 2026 18:23:00 -0700

The dream of LLM-powered ML workflow generation

At my company, an ML feature called provides capabilities like RFM analysis, recommendation, and contextual bandits. To run ML predictions at scale, the system calls an ML API that spins up parallel workers on AWS Batch behind the scenes. To make this parallelization work, input tables are aggregated per profile, which is a deliberate trade-off for scalability. These processes are orchestrated through digdag workflows (.dig files, executed on Treasure Workflow, a hosted digdag service) containing SQL (Hive or Trino), where the ML API is invoked via digdag’s http> operator.

Originally, the pre-processing and post-processing workflows were built by MLEs on a paid Professional Services team using their own templates, then deployed to customers who purchased PS engagements. But there was a desire to scale this beyond PS customers, and LLM-based workflow and SQL generation was seen as the path forward. Despite models getting better every day, generating stable workflows and SQL with LLMs proved difficult. For example, TD-specific UDFs in SQL don’t come naturally to the models. After several attempts, we had given up.

The rise of Claude Code and agent-friendly CLIs

Then our CEO called for a push toward becoming an AI-native organization, and Claude Code was rolled out company-wide. Adoption spread beyond software engineers to PdMs, Solution Architects, and even sales. Two developments were particularly impactful: , a unified agent-friendly CLI that can call TD’s various microservice APIs, and , a desktop application built on top of tdx. With Claude Code and tdx working together, agents could create marketing journeys, analyze tables in TD, and visualize results. The CEO personally created onboarding challenge tasks for employees to accelerate adoption, and as a result, automation has been spreading across both internal and customer environments.

Out of this momentum, a Skills marketplace was born for both and internal use. These skills improve reproducibility and accelerate automation of complex tasks.

What the agent-friendly CLI made possible

The biggest contribution of tdx was exposing Treasure Workflow’s endpoints through a CLI, which let Claude Code autonomously create workflows, push them, run them, inspect the results, and iterate. Workflows can take anywhere from a few minutes to over an hour to execute, which has always made automated testing painful.

Thanks to Claude Code + tdx, it became possible to generate a workflow, queue it for execution in the background, and verify the results. This was a game changer.

An agent-friendly CLI that handles API interactions end-to-end is no longer a nice-to-have. It’s essential.

Turning ML workflow templates into skills

That said, as I mentioned at the top, no matter how smart Claude Code’s models get, it’s still hard for an LLM to generate workflows that are consistently reliable for anyone to run, especially for customers with no prior knowledge.

So I reframed the problem. If generating workflows and SQL from scratch is too hard, why not create templates that generate workflows and SQL from configuration parameters? By embedding templates in a skill, the LLM’s responsibility narrows from “generate workflows and SQL from scratch” to “choose the right parameters for the data and the problem.” The workflows themselves become deterministic since they’re pre-built as templates. Deterministic logic should live in scripts, not in the LLM’s probabilistic output.

This idea was inspired by how cdp-api dynamically generates digdag workflows from database values.

(Japanese, see slide 29 for the code example)

Jinja2 templates for digdag workflows

Concretely, I templated the .dig files with Jinja2 and had the LLM focus on determining configuration values, with config.yml as the single source of truth for all modifiable parameters. Claude renders the templates directly from config.yml without any external tooling. Seeing the .dig.j2 extension for the first time gave me a small thrill.

Internally, I use two kinds of variables: render-time parameters with {{ }} and digdag runtime parameters with ${ }. The former handles things like branching on whether the SQL engine is Hive or Trino, or when algorithms and hyperparameter candidates can be fixed ahead of time. The latter is for cases like storing hyperparameter tuning results in a table, then dynamically assigning those values to a training task at runtime via SQL.

OpenAPI as the contract with the agent

One tricky part of templating was that the ML API accepts complex parameters, and somehow the agent needs to understand all of them. Fortunately, our project managed ML endpoint parameters with OpenAPI, so we could hand the full spec to the agent.

Our project uses to generate model.py from the OpenAPI spec for parameter validation at runtime. Giving the machine-readable openapi.yml to the agent and having it translated into a markdown document within the skill turned out to work great. Long live standard formats.

Skill-creator agent vs. skill-user agent

While building the skills, I asked Claude how best to test them. Its suggestion: spin up a separate agent process to exercise the skills through trial and error. I tried it, and it was an excellent experience.

When you’re using the skill-user agent, you’re not reading the OpenAPI spec or skill documentation yourself. Instead, you start thinking in terms of what you want to try: “I want to run this algorithm with this parameter combination.” Normally, when doing manual sanity checks, the spec is already in your head, and you tend to skip the tedious, complex parameter combinations. But when an agent can do it for you, you get greedy.

The skill-user agent came back and told me: “I looked at the skill, but that parameter combination isn’t supported in the OpenAPI spec yet.” I had assumed our QA end-to-end tests would have caught this, but it was a close call. Because the OpenAPI spec was maintained manually, the Python code internally supported the combination, but the spec was missing the parameter, so requests couldn’t pass through.

I fixed the bug quickly, deployed to the development environment, and updated the skill. Claude then picked up the new parameter combination and used it as if it had always been there. Impressive.

Building the skill alongside the product taught me that having a live execution environment pays for itself many times over.

Wrapping up

By sharing these skills on the internal marketplace, the workflow creation step that used to require paid PS engagements was simplified, and even customers without PS contracts could benefit.

Treasure Studio also helped here: ML prediction results can now be visualized directly, making it easy to run analysis and model improvement cycles. Turning those analysis patterns into skills too seems like a natural next step, but that’s out of scope for this post.

When I was drafting this post, I bounced ideas off Claude, and it argued that “the LLM’s strengths are understanding problem structure and parameter inference.” Providing an end-to-end CLI to support that lets agents run the generate-execute-fix loop autonomously. And by templating the complex parts of that loop, you create a clean division of labor: domain experts build the templates, and agents (used by people without that domain knowledge) fill in the parameters.

Migrated from Pages CMS to Sveltia CMS

Thu, 19 Mar 2026 16:41:00 -0700

I , but after encountering several concerns, I migrated to Sveltia CMS.

Motivation

As I wrote in , the move was triggered by the following issues with Pages CMS:

It handled time zones poorly (essentially forcing everything to +00:00), and even after , there didn’t seem to be much interest in fixing it.
It was quite troublesome to achieve the directory structure recommended by Hugoblox, where content (index.md) and images are placed in the same folder. I ended up having to upload images manually.

What I did

Deployed Sveltia CMS Auth to Cloudflare Workers for authentication.
Had Claude migrate the .pages.yml from Pages CMS to static/admin/config.yml.
Copy-pasted static/admin/index.html from the documentation.
Used Claude to restore the time zone data that Pages CMS had dropped.

The biggest hassle was deploying Sveltia CMS Auth, but since I was already using Cloudflare Pages, all I had to do was click the deploy button in the README at and follow the instructions. It was simple. It reminded me of Heroku.

For details, please refer to the following PRs:

Impressions

Pages CMS had been bothering me with a few minor annoyances, and it was great to see them resolved here. For instance, loading 400+ posts takes over 10 seconds in Pages CMS, but Sveltia CMS handles it in about 2. It’s fast enough, isn’t it?

The M↓ button is a lifesaver too. Being able to drop into raw Markdown when the editor misbehaves means I no longer have to open GitHub and edit files directly, which was a nightmare, especially on mobile.

The attention to detail really shows. While writing this post, a minor GitHub outage hit, and I actually got a warning about it. Impressive for something that runs entirely client-side.

Uploading this screenshot straight from the clipboard was seamless as well. In Pages CMS, getting a screenshot into the right folder alongside the content was a real pain, so this one stood out. (This feature . What a sense of speed!)

The author, kyoshino, is a Japanese speaker, it is clear that they are mindful of the IME input issues that we CJK (Chinese, Japanese, Korean) users often encounter. Being able to type without stress is truly important.

I plan to enjoy trying it out for a while, and unless any major issues arise, I think I will stick with it for the foreseeable future.

Between Principal and Glue Work

Sun, 08 Mar 2026 13:07:00 -0700

Introduction

I have always worked with the mindset that being a Staff+ Engineer is essentially doing high-level “ .” Because the tasks are surprisingly diverse, I have written about them , but I have never quite been able to cohesively organize my thoughts.

And for a while, I also struggled with how to effectively express the impact I’ve made on my resume. This is because the role requires catching things that fall through the cracks far more often than one might expect, and while delivering value through these tasks is incredibly important, resumes tend to demand flashy, shiny achievements (like ).

My stance has always been, “I will do whatever it takes to deliver a valuable product to the customer.” The scope of “whatever it takes” probably varies from person to person. So, as a starting point, let’s unravel this by comparing it with Will Larson’s , which are well-known to those familiar with .

The 4 Staff Archetypes

Let’s recap Will Larson’s four Staff Archetypes.

1. Tech Lead

A role deeply involved with a specific team (or a few teams), leading technical direction and execution.

Characteristics: Takes responsibility for the team’s technical decisions, fleshing out complex tasks, and unblocking progress.
Primary Activities: Focuses more on shaping the overall technical vision for the team, mentoring members, and coordinating with product managers rather than pure implementation. This is the most common archetype and a natural extension from a senior engineer role.

2. Architect

A role responsible for cross-organizational success and quality within a specific technical domain (e.g., API design, frontend, infrastructure strategy).

Characteristics: Formulates technical strategies spanning multiple teams and maintains long-term technical alignment.
Primary Activities: Deeply understands business needs and technical constraints to guide the overall architecture of the organization. Needed in large organizations or companies with complex systems burdened by accumulated debt.

3. Solver

A “firefighter” role that moves beyond a specific team to solve critical and difficult technical challenges for the organization.

Characteristics: Deployed to problems with high execution risk or complex issues where the solution is unclear.
Primary Activities: Goes to the frontline where the problem is occurring based on requests from leadership, and moves on to the next challenge once resolved. Requires pure technical breakthrough ability rather than organizational coordination.

4. Right Hand

A role acting as the “right hand” to executives like the CTO or VP, borrowing their authority to solve complex organizational problems.

Characteristics: Operates at the intersection of technology, business, people, and processes to permeate the executive’s intent throughout the organization.
Primary Activities: Attends executive meetings and helps remove organizational bottlenecks and execute strategies. A rare archetype found in massive organizations with hundreds of engineers.

However, Will Larson also states the following (quoted from ):

This taxonomy is more focused on being useful than complete, but so far, I’ve been able to fit every Staff-plus engineer I’ve spoken to into one of these categories. Admittedly, some folks are easier to classify than others.

I started writing this article because I couldn’t help but feel that I might be one of those “harder to classify” ones.

Breaking Down My Work by Archetypes

Let me briefly explain my positioning as a Principal Software Engineer or Tech Lead at my current workplace.

I didn’t have the official title of Tech Lead, but I operated as the engineer overall responsible for driving the engineering of the company’s ML product development. Regarding the organizational structure, it was reorganized about a year ago by the CTO and VPoE. I, as a Principal, and the Engineering Manager of the ML team existed as peers, reporting directly to the Sr Engineering Director who oversaw multiple engineering teams. This is what is often called a “Two-in-a-box” style.

Let’s break down the main things I’ve done over the past three years using the archetypes.

As a Tech Lead / Architect
- Consistently led the grand design, PoC, scale validation, and release of the ML training and prediction infrastructure (Python, FastAPI, AWS Batch), successfully releasing it with 2 people in 5 months.
- Scaled the RFM prediction processing, making it up to 100x faster and supporting processing for 1 billion users.
- Implemented the PoC for the recommendation ML solution model and selected scalable algorithms.
As a Right Hand
- Directly explained the engineering roadmap to the CTO and product leadership to secure sponsorship.
- Secured sponsorship for a complete revamp of the ML infrastructure over a year and drove it through to release.
- Drafted the product roadmap and proposed product direction not only to Engineering but also to the Product Manager and VPoP.
As a Solver
- When an escalation came from the PS team saying, “The customer needs this feature right now,” I jumped in as a firefighter, resolved the issue, and released it to production in 1-2 weeks.
- Rewrote incomplete code from a project where the engineer who originally worked on it had moved on, elevating it to production-grade.

As I was writing this, I realized I also paid attention to things like the following:

Task assignment considering members’ motivation and project planning with their careers in mind (e.g., taking on tedious tasks like building admin consoles myself and handing challenging tasks to members).
Creating draft UI design proposals to communicate the complexity of the data model to UX designers.
Fundamentally rewriting JDs and acting as a gatekeeper in interviews.
Providing indirect performance evaluation input to the EM during 1:1s.

In short, what I was doing was essentially ¹ . However, my focus was on catching things that fall through the cracks to prevent the project from failing and ensure its success, giving it my all according to their importance. Because of this, I often wondered what my actual responsibilities were, but reading Staff Engineer books out there made me realize that more or less everyone does this, so I gritted my teeth and kept unblocking things.

Why Does the Scope of Staff+ Expand?

Generally, the Two-in-a-box model is well known, although the breadth of role division varies. For example, in “ ,” Charity Majors writes about the collaboration between EMs and TLs:

There is an enormous demand for technical engineering leaders — far more demand than supply. The most common hackaround is to pair a people manager (who can speak the language and knows the concepts, but stopped engineering ages ago) with a tech lead, and make them collaborate to co-lead the team. This unwieldy setup often works pretty well.

Also, at , every team has an EM, and for many projects, a TL is placed separately to collaborate with the EM, or if there is no TL, a Tech Lead Manager holds both roles. In their case, the division of roles between EM and TL is as follows:

Engineering Manager primarily focuses on people management (staffing, coaching & growth) and organizational strategy (organizational risk, operational efficiency, team charter & outcomes)

Tech Lead primarily focuses on technical leadership (technical execution, technical strategy, technical culture, roadmap feasibility & execution).

In this way, the advantage of Two-in-a-box is sharing the cognitive load so that attention can be paid to various areas.

However, in domains requiring deep technical knowledge like ML, if there is a gap in the technical depth between the EM and the TL, it inevitably becomes difficult to make highly accurate technical decisions. As a result, the “division” turns into a broad “delegation” to the TL, and the scope of the TL expands. In my case, for instance, I was making decisions as a TL up to task assignment and resource allocation, being delegated some of the EM roles at Asana.

In my case, I felt the Two-in-a-box system worked better than expected. Especially after we clearly defined our boundaries and respective territories, we just ran autonomously. By fundamentally dividing roles—People Management for the EM and Technical Leadership for the TL—and reporting as peers to a common Sr Engineering Director, the expansion of the TL’s scope became organizationally acceptable. Before this structure was put in place, we frequently had minor clashes that looked like we were stepping on each other’s toes, which made me realize how important it is to build a good organizational structure.

Ultimately, if the EM and TL trust each other and have a system where they can work autonomously, it’s a trivial matter which way the border leans a bit.

Reflecting on My Career, or Redefining “Glue Work”

I’ve written a lot so far, but if I were to reorganize the work I’ve done and the scope I’ve expanded based on the Archetypes:

Technical decision-making and delivery as a Tech Lead/Architect
Proposing technical strategies to executives and gaining approval as a Right Hand
Critical problem-solving skills as a Solver
- Occasionally stepping into scopes beyond Staff+ (PdM, PjM, UX Designer, parts of EM) when necessary

It turns out that doing glue work as a Staff+ is not just doing random chores, but rather “carrying multiple Staff+ functions necessary for the organization all by oneself.”

Will Larson might be a little surprised too.

But well, I think someone who can move like this would be highly valued in a company with a strong startup culture.

To Those With the Same Struggles

If you ever feel like you’re losing sight of your core value while doing glue work as a Staff+ engineer, I recommend looking back at the Archetypes and verbalizing what you are actually achieving.

In the AI era, verbalizing allows for deeper exploration, so it might be good to bounce ideas off an AI to articulate your thoughts. I myself have been trying this after a former colleague of mine, , told me that dialoguing with AI for introspection deepens self-understanding and helps find directions for growth. It demands real concentration and can be draining, but the rewards are significant.

Also, to keep me sane, the case studies in books about Staff Engineers have been a great support. It would be good to keep both and close at hand. Reading them after a painful experience provides a deeply enriching reading experience that resonates more the more you digest it.

Doing glue work might just be a sign that you are thriving as a Staff+. However, never forget to prioritize your tasks by considering their business impact.

In the Japanese engineering community, there’s a term “高機能雑用” (kōkinō zatsuyō) — roughly “High-functioning general-purpose grunt work.” It originated from who, when asked by his director what he actually does, replied: “I do Hadoop, Hive, MySQL, machine learning, log analysis, NLP, VBA, and firefighting — but that’s too much to explain, so I just say ‘High-functioning general-purpose grunt work.’” That’s pretty much what being a Staff+ engineer feels like. ↩︎

Tackling Review Fatigue by Document Driven Agentic Coding

Fri, 19 Sep 2025 21:26:00 -0700

This year, the amount of time I spend on reviews has exploded. This applies to both code and documentation. And the fatigue from this has also increased dramatically.

Of course, this is because the speed and volume of output have increased with the help of LLMs. However, the quality hasn’t necessarily improved; in fact, I feel my productivity is declined.

I’m going to do a brain dump to write down what the challenges are and how I’m dealing with them.

Proxy Prompting Through Other Humans

I’ve been working as a tech lead on machine learning projects for a while now. As part of my job, I often write technical documents like , but I also have many opportunities to review them.

The difficult thing about machine learning is that when a project changes and the problem to be solved shifts even slightly, a completely different set of knowledge becomes necessary, requiring you to read new research papers.

Imagine you’ve been building web applications with Rails for years. Then one day, someone hands you the specification document for a Ruby parser generator, saying, “It’s Ruby, so it should be the same, right?” You can tell it was all generated by an AI, but how do you review it? How do you provide feedback to make it a sound specification? You would probably start by reading implementations in other languages or research papers to build a mental index before you could even begin the review.

Having to review specifications in such a world is truly exhausting. It’s a struggle just to get to a point where you have more knowledge than the generator. And after you’ve made the effort to provide appropriate comments, what awaits you are rebuttals made entirely by a generative AI, and the revisions are drastic, overwriting everything on a large scale without regard for the previous context.

I describe this situation as “prompting through others,” and frankly, in most cases, it’s faster for me to just read the papers and discuss them with an LLM myself.

Some readers might think, “Why not just have an LLM review the text generated by an LLM?” But this is fraught with a difficult problem: who guarantees the correctness of the LLM’s review? I often feel this even when I’m interacting with an LLM myself - “ .” ( ), or . The output is inconsistent; if you ask the same question three times, you’ll often get three different answers.

I believe that the limits of current LLMs and generative AI are defined entirely by the limits of human ability. If the person acting as a proxy continues to be just a proxy, they will become unnecessary. It’s important to escalate this appropriately to a manager.

As a side note, text generated by LLMs can often be detected by the high frequency of certain words or the . But as a non-native English speakers, it’s often given away by the presence or absence of quirks specific to a native speaker’s language. So, I’m trying to be confident in the results I generate and pass them on to others as our own opinions.

Lack of Self-Review, or the “Eyes Glazing Over” Problem

As I wrote in the previous section, it’s often the case that people don’t self-review what they’ve generated. This only breeds mistrust, so it’s best to stop immediately. However, the reality is that even with self-review, it’s easy for your eyes to just glaze over the content.

I learned in a 1-on-1 with my manager that there’s a tendency for people, even the same person, to be more lenient when reviewing code they generated themselves and stricter when reviewing code generated by others. I get it.

When I think about why this happens, it’s because non-hand-crafted code often doesn’t get stored in your brain’s cache. Furthermore, when generating code with something like Claude Code, the user experience makes it tedious to look at every diff. So, you end up generating something that works in “auto-approval mode” and then reviewing it. But when there are massive changes, you can’t keep up with the details. And you can’t to resist to think, “Well, the unit tests are passing, so it can be okay.”

Big Bang Commit

Recently, I received a pull request which diff was +10,000 -7,000 lines in a single commit. Yes, a big bang commit PR.

When you generate code with an LLM, especially in an auto-approval mode, the agent goes through a lot of trial and error. And a characteristic of agentic coding is that it overwrites large chunks of code aggressively. The existing design, which would have been respected if the same person were writing it by hand, was largely ignored in the PR generated by the LLM.

There’s a research suggesting that . I read a paper that . Yeah, that makes sense.

Also, LLMs don’t automatically break down tasks into appropriate granularity while Claude Code and GitHub Copilot create a TODO list. It’s quite difficult unless you give them careful instructions.

By the way, it remains an open question that making appropriately sized commits is a universal and very difficult problem, especially for exploratory implementations like in machine learning.

The Difference in Incentive Structures Between the Generator and the Reviewer

Many leaderships want to claim that “using LLMs increases coding productivity,” and I understand this is because many people strongly believe in AI-driven productivity improvements to prove its ROI, and there’s a hype that should be true.

Now, I’m aware there are various debates about code generation throughput, but I think it’s possible it will increase. However, the speed of the reviewer’s side hasn’t gotten any faster yet. It will probably take a little more time (at least a year?) before we can entrust all reviews to LLMs. Also, we can’t avoid an extra step to review LLM generated code.

We end up with an increase in a large amount of code that has been generated with a lenient self-review because the creator’s eyes glazed over. In other words, the generator’s goal is to maximize generation throughput, while the reviewer, regardless of that, must ensure quality. This difference in incentive structures is likely the main cause of review fatigue.

How to Approach Code Reviews Mostly Generated by LLM

So, what am I doing about it? Honestly, I haven’t found a silver bullet yet, but I want to share the results of my various experiments.

1. The Generator Reviews Their Own Code as Someone Else’s on GitHub

This comes from my own reflection on a time I made a rather large commit (+700 lines or so). Recently, in addition to reviewing and approving the code generated by the LLM in my local VSCode every time during implementation, I’ve also started reviewing the PR in a draft state on GitHub.

By doing this, my mindset switches to “reviewer mode,” and I’ve been able to find various flaws.

It may seem obvious, but I think using a dedicated review view is good because it puts you in the same mental model as when you’re reviewing someone else’s code.

2. Document-Driven Development

In the story of developing a Java version manager called (Rust based Java version manager , several helpful initiatives were taken, so I tried them myself.

The three important points I learned from the article and the actual commits are:

Always give the LLM a task size that a human can implement in about 30 minutes (to avoid context overflow).
Follow a flow of requirements definition -> external design -> work plan.
Furthermore, commit the above documents along with the code.

A very important point is that by committing the requirements document, external design document, work plan document, (and acceptance tests) as documentation along with the implementation, the intent becomes easier for humans to understand, even if the commit is somewhat large. Also, by properly organizing the work plan, you can provide the LLM with a rein to develop and commit at an appropriate granularity.

I tried this method myself by open-sourcing a script I had written as an internal tool.

I can’t show you the proper commits from when I was adjusting the internal code for open-sourcing, but the general workflow was as follows: (GitHub Copilot + Sonnet 4)

First, discuss what I want to do with Sonnet 4 in Agent mode and organize it into docs/requirements.md, docs/interface.md.
Have it create a TODO list with checkboxes for each phase of the implementation and save it as docs/plan.md.
Have it create an AGENTS.md file containing these documents and development conventions, toolsets, etc.
Implement each phase and commit. At that time, check off the item in plan.md and commit.
When moving to the next phase, clear the agent’s context and have it read AGENTS.md and plan.md to start the work.
When the planned phase is finished and it’s time to move to the next milestone, create a folder like docs/milestone1, move plan.md there, and make docs/plan.md an empty file to start development on the new milestone.

The actual PR for the open-source project looks like this:

Honestly, the commit granularity is large, and several features are committed together, but if I wanted to, I could easily make a commit for each checkbox because I’m managing the plan.md file.

In other words, the good thing is that you can understand if there are any major problems at the design level from the documents. If the design is flawed, you can go back and discuss it there without having to read the implementation, which I believe helps avoid pointless reviews.

How to Approach Document Reviews Mostly Generated by LLM

Honestly, I have no answer yet. What a colleague said really resonates with me: “People who blindly accept and pass on LLM-generated text are no different from the types who believe something is true just because a TV host said it on a variety show.”

Based on my experience, I’m discussing the feedback way and wording with Gemini. LLM supports creating the appropriate wording for a feedback/escalation and suggests escalation strategy as needed. (Gemini is a lonely Principal’s mentor!) Have it enumerate the problematic data points and summarize them. Summarization is inherently a strong suit of LLMs, and models tuned by American companies so they are good at American style of feedback.

Continue Struggling

I’ve written about various challenges and what I’ve done about them, but honestly, I’m still feeling my way through this. However, fundamentally, the limit of an LLM is the limit of the human using it. Therefore, it will still be necessary for humans to create workflows with a primary focus on how to maximize their own abilities.

Configured Pages CMS

Sun, 24 Aug 2025 15:11:00 -0700

I had been looking for a way to write Hugo articles on mobile devices like my iPad, and that’s when I came across this article by mehori and decided to try Pages CMS.

To be honest, it doesn’t properly support Hugo’s folder structure (/articlename/index.md), which makes image uploads a bit tricky. On top of that, I couldn’t upload images from my iPhone because of a 413 error. But, since I was able to set up and write text-only articles in Japanese from my iPad, I’ll count it as a win.

Along the way, I ran into a very minor YAML frontmatter parsing error, but I was able to find a workaround.

It’s a platform where you can figure things out by reading the code, and it has the bare minimum functionality, so it’s good enough for my needs.

Here are my current settings:

I also migrated from wowchemy to hugo-blox, which was quite a hassle. You can find more details in the PR. But seriously, the names have changed way too many times: Hugo Academic -> Wowchemy -> Hugo Blox…

I also fixed an issue where Amazon affiliate images weren’t showing up by changing the settings to not display images.

After the migration, I realized the search wasn’t working, but I managed to get it running by:

Adding a setting to create a pagefind index in the Cloudflare build command -> && npx pagefind --source 'public'
Trying various things with Cloudflare’s Rocket Loader, but it didn’t work, so I ended up disabling it.

Now it’s working somehow, but it seems like the Japanese search won’t work properly. Oh well, what can you do?

Machine Learning Project and Scrum

Fri, 02 May 2025 15:49:17 -0700

I’ve worked on several machine learning projects, and intuitively, I’ve felt that Scrum doesn’t seem well-suited for machine learning. However, during an internal discussion, a colleague said, “If we use Technical Stories, we should be able to break down tasks to fit within two weeks for any tasks. And if we do that, we should be able to deliver value in two weeks for ML products.” I couldn’t properly counter this, so I’m writing this article to articulate my thoughts, including how others in the world are approaching this.

Disclaimer

I have extensive experience in ML but is not particularly knowledgeable about Scrum, having only participated in a few projects as a development member.

Research Method

Brainstorming with Gemini 2.5 Pro (experimental)
Investigating experiences mainly on Reddit

Anticipated Challenges

Iterative development invariably occurs in ML projects. This is especially true because exploratory phases (EDA and model development) are always involved, and these phases generally involve rework.

I’ve always felt a strong incompatibility between these high-probability rework for ML tasks and the “deliver customer value in two weeks” sprint.

Using Technical Stories

What exactly are Technical Stories? I looked into it, and the example at the beginning of the following article resonated with me.

As a developer, I want an automated build So that I can be sure my code works.

I’ve actually seen stories like this here and there.

However, the article above, when considering where the business benefit (which I understand to be synonymous with customer value) lies, argues for the following template to ensure business value and incorporate features:

In order to will need .

Further refining this leads to:

In order to As a I want to <do something, or use or be restricted by some feature>.

In order to stop bots from spamming the site As a member of the commercial team, I want users to fill in a captcha box.

Happy ending? Really?

Business Value in ML Projects

The business value in ML projects and ML products remains providing something that positively impacts customers.

Achieving a model with good accuracy doesn’t immediately deliver customer value; it’s only when the prediction results can be served in the actual production environment that value is delivered to the customer. Alternatively, it might involve summarizing analysis results based on predictions and presenting actionable proposals in a report.

Conversely, I find it quite difficult to deliver such customer value with tickets spanning a few days to two weeks. Often, when a model is being developed, the deployment environment might not even exist yet.

One direction, as Gemini suggests, is to consider “risk reduction and uncertainty reduction as sprint goals.” While not direct customer value, there is certainly business value. Gaining insights like “this feature seems effective for prediction” or “this algorithm was unsuitable for this problem” is valuable in itself. That sounds reasonable.

However, by reading , I got a hint for the question of “Why it’s difficult to fit ML tasks into two weeks?”.

Differences in Task-Level Development Flow

compares the task progression in typical software development versus ML development based on the following diagrams.

It states that while task progression is linear at the task level in traditional software development, tasks in ML projects are often cyclical. (The following diagrams are both quoted from )

Linear tasks in traditional software development are also said to be “completion-oriented” tasks, where the goal is simply to complete the project.

Of course, ML projects also have linear tasks similar to traditional software development, such as data ingestion pipelines, basic preprocessing, developing demo applications with , and developing prediction APIs.

On the other hand, exploratory and experimental tasks are cyclical processes that involve repeating data understanding, hypothesis formulation, and verification. What’s difficult is that within a two-week sprint, there’s no guarantee of improved prediction accuracy, so nothing can be committed.

How to Implement Scrum in ML Projects

So, what should we do? Even so, some might be thinking, “Top-down, we’ve been told to do two-week sprints, so how do we implement it?”

Looking at several discussions on Reddit, I found the following two approaches:

1. Stop Using Scrum and Use Kanban

This might sound blunt, but adopting Scrum as the workflow isn’t mandatory for agile development.

“Agile Data Science with R” points this out in :

Both methodologies are applied with great success and it’s important to keep in mind that they are a means to an end, not religions. The Agile values and principles should be the primary guideline and when selecting one of the workflows you do so because it is the best way to work in an Agile way because its the best fit for the given situation.

The point is that Scrum or Kanban isn’t a matter of religions; what’s important is for the team to choose the best option to achieve their goals and to monitor whether it remains the best fit.

The author’s company initially adopted Scrum due to the many experienced Scrum practitioners, but it didn’t work well for model development, and they eventually moved to Kanban ( ).

The reasons cited include that Scrum is too rigid and lacks flexibility for exploratory and experimental phases. Also, as mentioned earlier, it’s impossible to commit to model prediction performance improvements without seeing the data. Therefore, there’s a suggestion that time boxing is better than using story points.

Scoping for data science is then not just estimating how long a task will take to complete, it is also time boxing. If used in this way, the scoping should be done in time units, not in a subjective measure such as story points. The data scientist should not take longer for the task than the team agreed upfront, wrapping up even when he does not feel completely finished.

Since Data Scientists tend to continue experimenting until they are satisfied, setting time limits rather than strict completion criteria is more rational in the research phase.

In practice, they seem to have adopted a Kanban board with the following six lanes. However, for some tasks, they ended up stopping after confirming that the model accuracy didn’t improve during the hypothesis testing phase.

to do
test hypothesis
code review hypothesis
update model
code review update model
done

A very similar point (use Kanban, do time boxing) can be found on .

Don’t confuse agility with solely scrum and its sprints, which are the root of the problem and work poorly in research mode.

This specific example is easy to understand:

Example: build a PoC in a week. If AUC exceeds X then it’s promising and let’s spend another 3 months on further extensions (data, features, architecture, hyperopt) and putting all into production. If there was no AUC gain on the last week, we do not extend any further. Inside this 3 month time box - execute pure Kanban, task by task, which allows you to take different paths as needed (agility), not waiting till your sprint finishes in 3 days. You already know your new feature is poorly designed and you need to start on tweaking it right now.

In other words, time-box, test hypotheses within that time, and if successful, decide to continue further extensions.

Another key point the author of “Agile Data Science with R” proposes is to quickly create an MVM (Minimum Viable Model) instead of an MVP. For example, initially deploy a simple model like linear regression to a limited set of users as an MVM, then add features, and finally deploy a more complex model (Random Forest is mentioned in the text, but now a NN-based approach might also be considered).

This connects with the idea of time-boxing mentioned earlier. Using an MVM as a hook is a good idea as part of an effort to prune the numerous options and find a high-probability path.

2. Use Hypothesis Based Stories (Modifying Scrum)

Another discussed an approach that mixes Scrum and Kanban. They use Scrum for long-term projects and time boxing for research tasks.

Specifically, they seem to have made the following changes to Scrum:

hypothesis based stories (instead of user based)
foregoe stand ups, people keep their tickets as research logs and @ people when they need help. - Product owner can read the tickets if they want to know where we are
monthly retro rather than per scrum, wider focus
tickets largely written by the data scientists then priorities by product owner

Skip stand-ups and have people keep research logs in their tickets, mentioning others in the tickets when they need help. The Product Owner can proactively check the tickets for updates. Data scientists largely write the tickets, and the Product Owner prioritizes them. The goal is to allow DS/MLE to focus on their work.

Hypothesis Based Stories are particularly unique. If we recall the discussion about business value in Technical Stories, the business value in the hypothesis testing phase of ML can only be described as “reducing risk and uncertainty.” Also, by making “formulating and testing a hypothesis” the goal of a ticket, it creates an awareness of the need to formulate proper hypotheses, and the outcome can be closed as “hypothesis was correct/incorrect.”

The point is that the exit criteria is provable and the delivery is typically the proof. Likewise, disapproving the hypothesis is still a success, we learned something.

Here are some examples of such stories:

“We believe a fine tuned distilbert architectures will allow us to identify cases with a precision of greater than .95 and a recall of greater than .7.”

“We believe it should be possible to transform a given article within our dataset to our standardised form without additional datasets or augmentation”

“We believe x metric can best be explained to stakeholders using a combination of shap values and distribution charts”

Aside: What Does a Product Owner Do in ML Projects?

As an aside, there’s a detailed chapter on what a Product Owner does if they don’t create tickets ( ).

In summary:

Takes on all communication with stakeholders, allowing DS/MLE to focus on tasks.
Helps discuss and organize the tasks created by DS/MLE through scoping.
Prioritizes the tasks created by DS/MLE.
Points out business concerns that DS/MLE might not be aware of.

Reading this, I feel like the time of DS/MLE is considered more valuable than that of SWE.

Conclusion

After a quick investigation, I’ve reached the following conclusions:

Strict Scrum is not suitable for ML projects.
If possible, separate workflows for research/exploration phases and development phases.
In the research/exploration phase, use time limits instead of story points.
In the research/exploration phase, proceed based on hypothesis testing (write Hypothesis Based Stories, use Kanban).

The idea of time boxing and progressing research tasks centered around hypothesis testing, while common sense in ML projects, is a very powerful support.

So, in response to the initial statement, “If we use Technical Stories, we should be able to break down tasks to fit within two weeks. And if we do that, we should be able to deliver value in two weeks,” my reply would be:

Understand business value as “reducing uncertainty.”
Use Hypothesis Based Stories instead of Technical Stories, or Kanban.
Instead of setting concrete goals within a two-week timeframe, quickly iterate through possibilities by setting time limits.

Migrated From Netlify to Cloudflare Pages

Fri, 02 Feb 2024 16:42:14 -0800

Netlify is a great service, but it is also known as slowness in Japan. I have been using Netlify for my blog hosting for a long time, but I decided to migrate to Cloudflare Pages to improve the speed of access to my blog from Japan.

The migration step from Netlify is pretty straight foward. I just need to follow this official guide: .

My blog is built by Hugo and I use Hugoblox, f.k.a., Wowchemy, as a theme. And, I manage my blog content on GitHub, and just adding the Cloudflare Pages app to my GitHub repository, and it automatically detects the settings and builds the site.

If you use Cloudflare for DNS, it automatically sets up the DNS settings for you.

The special consideration on build settings is that I need to set the environment variable HUGO_VERSION to the version of Hugo that I use. In my case, I use Hugo 0.88.1, so I set HUGO_VERSION to 0.101.0. Also, I need to set -b URL option, it was $URL in Netlify, but it is $CF_PAGES_URL in Cloudflare Pages.

The build time is pretty fast, and the PageSpeed Insights score is also improved. I can feel faster access on my browser as well. I’m happy with the migration. Actually, the major reason of slowness was downloading fonts and using Font cache on Cloudflare solved the problem.

Scrape Notion and convert into PDF

Fri, 26 Jan 2024 17:40:00 -0800

I love , who is a Japanese meal kits provider in Vancouver. Their meal kits are really tasty and authentic Japanese foods. I can’t live without them. When I visited Japan last year, I wasn’t too eager to find nice Japanese restaurants because of them.

Recipe on Notion is good, if it’s printable

They provide a recipe on Notion. Seeing the recipes on it is great since they can fix recipes quite quickly.

However, there’s one caveat of Notion. They don’t provide printable pages. It’s super annoying to copy and past the recipes to the memo app, and print it out. I asked Notion’s support team, but they answered it isn’t a prioritized item implicitly.

Ok, it’s automation time!

Scrape Notion with Python

As my handy tool, I have been using Python for this kind of automation for years. Originally, I used beautifulsoup, which is great package for web scraping, but I gave it up to use it. Contents of Notion is rendered by JavaScript dynamically.

I chose and it works like a charm.

Here is the GitHub repository:

They key takeaways are:

chromedriver-autoinstaller package is useful to avoid extra efforts of Chrome driver installation.
Selenium is easy enough to export PDF .
Running the script on GitHub Actions is easy. Don’t forget to install fonts if it’s not English page.

Originally, I thought I had to prepare a Docker image, but I was aware it was not mandatory. Managing a Docker image for this kind of hobby script would be costly. So, I’m going to keep this approach and will look back if it is the right way.

Currently, I scheduled the . It will update the PDFs on the repository automatically.

Edit: Now I use Cloudflare Pages to host the PDFs. You can check at .

No Python environment on a local machine is needed anymore.

Yay, automation is completed! 😁

tabula-py 2.8.0 now uses jpype to launch JVM

Sat, 09 Sep 2023 17:13:08 -0700

Recently, I released tabula-py 2.8.0. It is a major release because it uses to launch JVM. This means that it reduces JVM launch time since jpype reuse JVM via JNI.

How fast is it?

I measured read_pdf_with_template function execution time, which repeatedly launches Java process in the previous version.

The example template contains 4 rules, which means it calls tabula-java 4 times.

$ cat examples/data.tabula-template.json | jq
[
 {
 "page": 1,
 "extraction_method": "guess",
 "x1": 153.99985500000003,
 "x2": 565.5698550000001,
 "y1": 123.999615,
 "y2": 531.7446150000001,
 "width": 411.57,
 "height": 407.745
 },
 {
 "page": 2,
 "extraction_method": "guess",
 "x1": 153.99985500000003,
 "x2": 453.879855,
 "y1": 123.99884999999993,
 "y2": 210.44384999999994,
 "width": 299.88,
 "height": 86.44500000000001
 },
 {
 "page": 2,
 "extraction_method": "guess",
 "x1": 153.99985500000003,
 "x2": 487.53985500000005,
 "y1": 410.99625000000003,
 "y2": 497.44125,
 "width": 333.54,
 "height": 86.44500000000001
 },
 {
 "page": 3,
 "extraction_method": "guess",
 "x1": 153.99985500000003,
 "x2": 235.85485500000001,
 "y1": 123.99885000000012,
 "y2": 322.8988500000001,
 "width": 81.855,
 "height": 198.9
 }

The result is as follows:

v2.7.0:

$ python -m timeit 'import tabula; tabula.read_pdf_with_template("examples/data.pdf", "examples/data.tabula-template.json")' 2> /dev/null
1 loop, best of 5: 1.31 sec per loop

v2.8.0:

$ python -m timeit 'import tabula; tabula.read_pdf_with_template("examples/data.pdf", "examples/data.tabula-template.json")' 2> /dev/null
1 loop, best of 5: 75 msec per loop

It is 17 times faster than the previous version!

Caveats

Since , you can pass java_options for the first time only. If you want to change java_options, you need to restart Python process.

Challenges for releasing v2.8.0

I had to solve several challenges to release this version.

The test issue with different `java_options`

As I mentioned, jpype doesn’t allow to reboot JVM. This causes unit test with different java_options to fail. I solved this by separating run with nox session.

See for details.

This limitation is not a big deal for tabula-py users because tabula-py users don’t need to change java_options frequently.

Read the docs default behavior change

Read the docs changed the default installation packages for Sphinx. I didn’t declared the dependency for Sphinx, so it caused the build failure.

The default behavior of RTD was just installing the latest version of Sphinx and sphinx-rtd-theme, however, now it installs very old version of them like:

I solved this by pinning the versions of dependency for Sphinx and sphinx-rtd-theme.

4 Steps to Release a CLI in Python

Fri, 20 May 2022 23:32:41 -0700

This is what I learned from creating a Python CLI ( ) in a day.

In just 4 steps, you can release a CLI written in Python easily.

Create a project by using poetry

is a modern Python packaging and dependency management tool. Poetry is becoming popular and defacto rapidly.

By using Poetry, it enables us to manage package dependency, to create a project template, and to publish to PyPI.

To setup a project with Poetry, this article is the best to read even if you build a CLI.

(originally written in Japanese)

One thing I added to my project is . isort is to sort imports automatically.

Here is the example of my project.

[tool.taskipy.tasks]
test = { cmd = "pytest tests", help = "runs all unit tests" }
pr_test = "task lint"
fmt = { cmd = "black tests digdaglog2sql && isort digdaglog2sql tests", help = "format code" }
lint = { cmd = "task lint_black && task lint_flake8 && task lint_isort && task lint_mypy", help = "exec lint" }
lint_flake8 = "flake8 --max-line-length=88 tests digdaglog2sql"
lint_mypy = "mypy tests digdaglog2sql"
lint_black = "black --check tests digdaglog2sql"
lint_isort = "isort digdaglog2sql tests --check-only"

Create a CLI with Click/Cloup

is a famous Python package to build a command line tool. You can easily create a CLI by using decorator.

Here is the example from the Click website:

import click

@click.command()
@click.option("--count", default=1, help="Number of greetings.")
@click.option("--name", prompt="Your name",
 help="The person to greet.")
def hello(count, name):
 """Simple program that greets NAME for a total of COUNT times."""
 for _ in range(count):
 click.echo("Hello, %s!" % name)

if __name__ == '__main__':
 hello()

is an extension of Click.

Using by Cloup, you can handle option groups and complex constraints like mutually_exclusive as:

@option_group(
 "Cool options",
 option('--foo', help='This text should describe the option --foo.'),
 option('--bar', help='This text should describe the option --bar.'),
 constraint=mutually_exclusive,
)

Constraints of Cloup can validate the dependency and it also renders constraints in help.

Use poetry-dynamic-versioning for version management

is a Python package to do same thing as . You don’t need to write version number by hand since this package use the version from tag of Git, e.g., “v.0.1.0”.

Managing version by Git enables you to release to PyPI from GitHub Actions. This means you can release to PyPI on mobile device by releasing from GitHub.

After installation of poetry-dynamic-versioning, you just add three thing in pyproject.toml:

[tool.poetry]
version = "0.0.0"

[tool.poetry-dynamic-versioning]
enable = true

[build-system]
requires = ["poetry-core>=1.0.0", "poetry-dynamic-versioning"]
build-backend = "poetry.core.masonry.api"

Note that build-system configuration may vary depending on how you install poetry-dynamic-versioning. See the document for detail.

Introduce GitHub Actions to release the package to PyPI

As I mentioned above, I highly recommend to use GitHub Actions to release a Package to PyPI.

Since GitHub provides now, creating a release from GitHub with triggering PyPI release is the best way to publish a new version.

Here is the snippet of GH Actions to release to PyPI by using poetry.

name: Upload Python Package

on:
 release:
 types: [created]

permissions:
 contents: read

jobs:
 deploy:

 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v3

 steps:
 - uses: actions/checkout@v3
 - name: Set up Python
 uses: actions/setup-python@v3
 with:
 python-version: '3.x'
 - name: Install dependencies
 run: |
 python -m pip install --upgrade pip
 pip install poetry
 - name: Build and publish package
 run: |
 poetry version $(git describe --tags --abbrev=0)
 poetry build
 poetry publish --username __token__ --password ${{ secrets.PYPI_API_TOKEN }}

Note that, while PyPI API Token can be found on PyPI, if you need to create project scope token, you need to upload a package manually first.

Create data lineage from Trino/Hive queries in digdag log with Python

Thu, 05 May 2022 20:31:05 -0700

What’s data lineage?

Data lineage is something to describe “Where this data comes from and where it goes?”

I learned this term in my previous job. They provided “Cloudera Navigator” which includes data lineage from execution logs of Hive/Spark etc.

lineage of Cloudera Navigator via https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cn_lineage_generation.html

sqllineage is awesome open source tool for visualizing lineage

Recently, I learned there is a Python package so called sqllinage, that makes analyze and visualize data lineage from SQLs.

sqllineage consists of Python implementation to analyze SQL and web application written in React.

Visualize data lineage from Treasure Data’s workflow logs

I found that Treasure Data’s workflow log outputs SQLs in its log. But it still needs to format pure SQLs.

Then, I create digdaglog2sql to extract SQLs from Treasure Workflow logs.

You can use it with Python 3.7+. Here is the overview of the usage and check details on GitHub.

Install via pip:

pip install --user digdaglog2sql

If you have a workflow log downloaded from Treasure Data, you can convert into SQL as:

digdaglog2sql --input workflow-log.txt --output output.sql

Or, if you want extract SQLs from specific workflow, you can use Session ID of it.

export TD_API_KEY=1234XXX/YYYYYYYY
digdaglog2sql --session-id 12345 --site us --output output.sql

You can fetch SQLs from your hosted digdag as the following:

digdaglog2sql --session-id 12345 --endpoint digdag.example.com --output output.sql

~~Note that, as of May 5, 2022, sqllineage and sqlparse, which is an important backend of sqllineage, are not fully compatible with Trino and Hive queries.~~

Note

As of 2022/05/11, the issues in sqllineage around Hive/Trino were fixed and it is available in 1.3.5 on PyPI. It means, you don’t have to have node for sqllineage installation from source.

As of 2022/10/06, the issue in sqlparse was resolved in 0.4.3.

~~These are the PRs that approaches the issues:~~

✅ -> Released in 1.3.5
✅ -> Released in 1.3.5
✅ -> Released in 0.4.3
✅ -> Released in 0.4.3

~~Don’t worry about it. I prepared patched branches on GitHub. You can install sqllineage and sqlparse as the following:~~

pip install git+https://github.com/chezou/sqlparse.git@trino#egg=sqlparse==0.4.3.dev0
pip install sqllineage

~~If you see some error on installation of sqllineage, double-check if you have node installed.~~

Then, you can visualize your SQL file as:

$ sqllineage -g -f output.sql
 * SQLLineage Running on http://localhost:5000/?f=output.sql

Now you can see visualization of data linage, both table level and column level.

An example of SQL lineage

Let’s try sqllineage!

3 configs add recommend articles into your Hugo blog by GitHub Actions

Tue, 25 Jan 2022 19:37:52 -0800

Hugo has a feature to show keyword based related articles.

Yeah, keyword based articles might be useful, for people who can manage keyword, category, etc, constantly. I’d love to add content based recommendation that doesn’t require to write explicit keywords by myself. Then, I found an open source named “Prelims” which is developed by .

Prelims is a post-processing tool for Front matter of Hugo/Jekyll, that is a metadata of an article. The recommendation method which is implemented for now is classical, create a TF-IDF based word vector and find similar articles by consign similarity.

The reason why I love Prelims is it’s simple and flexible. Post-processing of front matter doesn’t break your articles nor blog system at all. You can remove extra meta data Prelims generated whenever you want.

Isn’t it practical, right?

One downside of Prelims is it requires to implement Python code for tokenizing or vectorizing TF-IDF. I don’t want to bring my laptop for blog writing and wanna use Netlify CMS and iPad without having Python environment.

So, I built a CLI tool for Prelims, named , which enables to add recommended articles just writing 1 configuration YAML file. It also runs with GitHub Actions.

The three things you need to prepare are:

Configuration YAML file for prelims-cli. e.g., scripts/config/myconfig.yaml
Hugo HTML partial layout, e.g., layouts/partials/page_related.html
GitHub Actions workflow for prelims-cli

Here is the example gist what you need to write.

where content/blog is the directory for English articles and content/post is the directory for Japanese articles.

Putting three files enables you to show recommended articles into your Hugo blog, like the screenshot in the top of this article.

Internally, for Japanese tokenization, it uses SudachiPy. Since keywords prelims generates are a-bit noisy and didn’t wanted to cleanup, so I stopped using it.

The good things I feel are, I can use my blog articles for my hobby recommendation project, and I don’t need to manage tags and categories seriously.

You can enjoy your recommendation without having Python environment, so you can write your articles on iPad with Netlify CMS!

py> operator development guide for Python users

Thu, 05 Mar 2020 14:15:52 -0800

How to build & test custom scripts on local env before pushing

General strategy:

Make a Python task reasonable granularity to run on local env

Since Treasure Workflow doesn’t have intermediate storage between tasks, a task can be huge. Considering container launch time, it would be better to create a single huge task, but it makes difficult for debugging. Starting from creating a reasonable size of function which is able to debug easily. Then, you can create a function that calls those minimal functions at once.

There are few options to develop py> operator on the local environment.

Use TD docker image
Create a Python virtual environment on local env

1. Use TD docker image

To develop a single py> operator task, you can use the official docker image to run python tasks locally. Like ordinal Python script, you can add the main guard like:

if __name__ == "__main__":
 your_function("default_argument")

As of Mar. 5, 2020, our latest official images are shown as the following:

digdag/digdag-python:3.7
digdag/digdag-anaconda3:2019.03

If you want to run a debugger toward Docker container, we recommend using PyCharm to run a remote debugger. See also .

2. Create a Python virtual environment on local env

Python provides venv to create virtual environments, you can create the same environment by using pip.

Download requirements.txt and constraints.txt from and you can install dependencies as same environment with digdag-python:3.7 as:

$ python -m venv .venv
$ source .venv/bin/activate
(.venv)$ pip install -r requirements.txt -c constraints.txt`

Using this virtual environment, you can develop by using the same packages on the local environment.

Note that this approach can’t ensure OS differences, which means the production environment is running on Debian but the development environment might be Windows/macOS X. This causes errors when executing OS-dependent commands like apt-get.

If you want to create the same environment with anaconda image, you can download environment.yml from , and run:

conda env update -n base -f environment.yml

Now you have the same Python packages with digdag/digdag-anaconda3:2019.03

Note that this command will overwrite existing conda environment, we highly recommend to modify name in environment.yml from base to your environment name like my-env, and run:

conda env create -f environment.yml

Test a workflow including Python

If you want to run an entire workflow on the local environment, ~~you can use~~ .

Warning

As of Mar 5, 2020, Treasure Data uses digdag v0_10 branch, but it may change in the near future.

Warning

As of Feb 14, 2021, Treasure Data moved to v0_11 branch. You may use the latest release branch.

Passing Parameters to py> operator

There are two ways to pass parameters into py> operator:

ordinal digdag argument
environment variable
digdag variable

1. digdag argument

Assuming we have a Python script named py_scripts/examples.py like:

def print_arg(msg):
 print(f"Message is {msg}")

Passing msg argument from simple_with_arg task can be like:

+simple_with_arg:
 py>: py_scripts.examples.print_arg
 msg: "Hello World"
 docker:
 image: "digdag/digdag-python:3.7"

If you want to pass multiple arguments, you can add arguments in your function, then add them into digdag arguments as well.

Note that digdag arguments can be passed into Python seamlessly so that you might face unintended variables passed by using keyword arguments **kwargs.

For example, in this case, docker variable can be passed as a dictionary {“image”: “digdag/digdag-python:3.7”}. We recommend having implicit arguments on a Python function.

Note that there might be unintended conflicts between digdag and py> operator. Assuming you set some digdag variables like the following:

_export:
 td:
 database: my_db

+simple_with_arg2:
 py>: py_scripts.examples.print_arg_td
 msg: "Hello World"
 docker:
 image: "digdag/digdag-python:3.7"

having python function print_arg_td with td argument like the following:

def print_arg_td(msg, td=None):
 print(f"'msg' is {msg} and 'td' is {td}")

In this case, td variable never can be None since exported td variable, i.e., {“database”: “my_db”} always should be passed. This may cause type mismatches like dictionary and string. We recommend avoiding to use preserved arguments for digdag, like td variables like:

td.endpoint
td.apikey
td.use_ssl
td.proxy.enabled
td.proxy.host
td.proxy.port
td.proxy.password
td.proxy.user

Note that these variables might be changed in the future. There are build-in digdag variables. See digdag build-in variables at

Also, digdag might converts unintended type e.g., an integer from a string, so we recommend to evaluate or explicitly convert type on a Python function.

2. environment variable

Environment variables can be another option to pass parameters to py> operator. An environment variable is reasonable for passing secure information or secrets.

For example, if we have a task simple_with_env

+simple_with_env:
 py>: py_scripts.examples.print_env
 _env:
 MY_ENV_VAR: "hello"
 docker:
 image: "digdag/digdag-python:3.7"

This MY_ENV_VAR can be accessed by using os.environ like:

import os

def print_env():
 print(f'Env var is {os.environ["MY_ENV_VAR"]}')

Using an environment variable should be important especially when you need to use secrets information e.g. Treasure Data API key or AWS secrets key, etc.

digdag has a feature to store secrets information. Secrets are stored on digdag (or Treasure Workflow) database when executing td workflow secrets subcommand.

Assuming you’ve set a secret named td.apikey. This secret can be passed to py> operator like:

+simple_with_env2:
 py>: py_scripts.examples.access_td
 _env:
 TD_API_KEY: ${secret:td.apikey}
 docker: image: "digdag/digdag-python:3.7"

from py_scripts/examples.py like:

import os

def access_td():
 apikey = os.environ["TD_API_KEY"]
 # Do awesome execution

If you try to pass secrets from ordinal digdag arguments, secrets will never be fetched from secrets DB. For example, if you have a task like the following:

+simple_with_env_ng:
 py>: py_scripts.examples.access_td_ng
 apikey: ${secret:td.apikey}
 docker: image: "digdag/digdag-python:3.7"

by using the following script like:

def access_td_ng(apikey):
 print(apikey)
 # Always shows "${secret:td.apikey}" insted of actual API key like "1234/XXXX"

3. digdag variable

If you want to read digdag variable in a Python script, you can use digdag.env.params as the following:

def read_workflow_env(msg):
 import digdag
 print(digdag.env.params["my_msg"])

Note that import digdag can be run only when the script is run as a digdag py> operator task. If you want to avoid import error, you should write try-except syntax like:

try:
 import digdag
 digdag.env.store({"feature_query": feature_query})
except ImportError:
 pass

Directory structures

I recommend having the following directory structure.

my_project
├── README.md
├── config
│ ├── params.test.yml <- Configuration file for run through test. Mirror params.yml except for `td.database`
│ └── params.yml <- Configuration file for production
├── awesome_workflow.dig <- Main workflow to be executed
├── ingest.dig <- Data ingestion workflow
├── py_scripts <- Python scripts directory
│ ├── __init__.py
│ ├── data.py <- Script to upload data to Arm Treasure Data
│ └── my_script.py <- Main script to execute e.g. Data enrichment, ML training
├── queries <- SQL directory
│ └── example.sql
├── run_test.sh <- Test shell script for local run through test
└── test.dig <- Test workflow for local run through test

You can generate this structure from a template by using cookiecutter-digdag.

How to install Python packages / OS packages

For installation of Python packages, you can use os.syste or subprocess.run like:

import os, sys
os.system(f"{sys.executable} -m pip install --upgrade pytd==1.4.3")

import subprocess
# arguments should be passed by list
subprocess.run([sys.executable, "-m", "pip", "install", "--upgrade", "pytd==1.4.3"])

Ensure you set the version number of Python package.

To install OS packages, you can execute like the following:

import os
os.system("apt-get update") # Need to run before doing apt-get install
os.system("apt-get install -y wkhtmltopdf")

How to read/write tiny variables between digdag tasks

To read a digdag variable, you can use digdag.env.params as mentioned above.

To pass variables to another Python task, you can use import digdag.

def store_workflow_env(msg):
 import digdag
 digdag.env.store({"my_msg": msg})

This example code sets my_msg variable which is able to use the following tasks like:

+store_msg:
 py>: py_scripts.examples.store_workflow_env
 msg: "Hello World"
 docker:
 image: "digdag/digdag-python:3.7"

+restore_msg:
 echo>: ${my_msg}

Error notification with Python stack trace

digdag has _error: syntax to send a notification for an error message. You can access ${error.message} digdag variable to send the notification for Slack or Email.

Assuming that if we have the following workflow:

+simple_raise_error:
 py>: py_scripts.examples.error_sample
 docker:
 image: "digdag/digdag-python:3.7"

_error:
 echo>: ${error.message}

with this Python script:

def error_sample():
 int("a1234") # raises ValueError

This script always raises ValueError and the workflow log shows stack trace of Python as the following:

2019-12-24 23:06:32 +0900 [INFO] (0039@[0:python]+simple^error): echo>: Python command failed with code 1: invalid literal for int() with base 10: 'a1234' (ValueError)
 from Traceback (most recent call last):
 from File ".digdag/tmp/digdag-py-2-1815457087076518360/runner.py", line 165, in <module>
 result = callable_type(**args)
 from File "/private/var/folders/y9/bnjb3krn39s22rmg_wvlnf7m0000gp/T/digdag-tempdir2111531196420040503/workspace/1_simple_1_2_2945225080250994454/py_scripts/examples.py", line 5, in print_arg
 int("a1234")
 from ValueError: invalid literal for int() with base 10: 'a1234' (runtime)

In this example, we use echo> operator to show the error message, but you can use mail> operator for sending Email or http> operator to send a Slack message.

How to release Python package from GitHub Actions

Mon, 25 Nov 2019 08:42:11 -0800

Photo by on

Recently, I changed my CI from Travis to GitHub Actions. GitHub Actions is handy and useful for testing, publishing Python packages.

Testing Python code on GitHub Actions

Migration from Travis is super easy, just writing a simple workflow like:

The benefits of GitHub Actions for Python are:

We can use build matrix (e.g., OS and Python versions) like Travis
Launch time of GitHub is faster than Travis
Easy for additional dependency installation by using uses syntax, which uses another workflow

For example, installing JDK can be written as:

- uses: actions/setup-java@v1
 with:
 java-version: '12'
 java-package: jdk
 architecture: x64

The downside of GitHub Actions are:

Unable to
Hard to find the resources for debugging on the internet and unable to ssh to the instance

Releasing Python package from GitHub Actions to PyPI

I created the workflow like the following sequence:

Push a tag from local, or create a tag on GitHub. Using enables you to make a new version from Git tag
GitHub Actions creates GitHub release from the tag
GitHub Actions publishes wheel to PyPI by using PyPI API Token

You can see the actual workflow on GitHub:

The key points are:

Triggering the workflow from Git tag

on:
 push:
 tags:
 - 'v\*'

2. Adding dependency for deploy task

deploy:
 needs: release

needs syntax supports to write dependency. In this case, I describe release job for creating GitHub release, and then deploy job publishes the package to PyPI.

3. Preparation secrets for PyPI

Recently, PyPI provides API tokens for package publishments so that you can get an API token for the specific project. See details on the official document since it is under beta, and spec might change.

After getting API Token from PyPI, you can set secrets on GitHub by clicking “Settings” -> “Secrets” on the project page. Using my example workflow, you should set __token__ for PYPI_USERS , and a token starting with pypi- got on PyPI configuration for PYPI_PASSWORD .

Now, you can publish Python package to PyPI by just tagging on GitHub.

How to test a new Docker image for digdag workflow on CircleCI?

Sat, 05 Oct 2019 13:17:30 -0700

Photo by [Campaign Creators](https://unsplash.com/@campaign_creators?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)

Photo by on

Testing workflow runnability would be important when we build a complex workflow. is a workflow engine which syntax is simple and is able to run tasks with SQL, Python, Ruby, shell script, etc. digdag has Docker executor and it works like a charm with py>, rb>, and sh> operators.

How to ensure a new Docker image runnable with existing digdag workflow? I’ll show the way to run through it on CircleCI.

You can see the example repo on GitHub:

An issue with digdag Docker executor on CircleCI

Although CircleCI docker executor is the primary choice for CircleCI 2.0, which easily run with arbitrary Docker image, since it launches remote sibling docker container. Hence digdag Docker executor assumes to mount a volume, like -v /tmp:/tmp, you need some workaround to avoid it.

In this article, I’ll show you how to execute local mode digdag, a.k.a. didgag run, on CircleCI with digdag docker executor.

Use CircleCI machine executor

tl;dr, use CircleCI , which runs VM on CircleCI.

version: 2jobs: test: working_directory: ~/app machine: image: ubuntu-1604:201903-01 docker_layer_caching: true steps: - checkout - run: name: Install digdag command: | curl -o ~/bin/digdag --create-dirs -L "https://dl.digdag.io/digdag-latest" chmod +x ~/bin/digdag echo 'export PATH="$HOME/bin:$PATH"' >> ~/.bashrc - run: name: Test digdag run command: | set -x digdag run test.dig

Machine executor has Python, Ruby, Java, and Docker CE by default, so you can easily run digdag on CircleCI.

Here are the dig file and Python script.

# test.dig+task: py>: test.show docker: image: "python:3.7-slim-buster"

Python script:

# test.pydef show(): print("Hello CircleCI")

Build custom Docker image and test with digdag

In some cases, you want to test whether a new Docker image works appropriately with existing workflow.

If you build a new Docker image for digdag Docker executor and test with existing workflow, you can write like the following:

version: 2jobs: build_and_test: working_directory: ~/app machine: image: ubuntu-1604:201903-01 docker_layer_caching: true steps: - checkout - run: name: Install digdag command: | curl -o ~/bin/digdag --create-dirs -L "https://dl.digdag.io/digdag-latest" chmod +x ~/bin/digdag echo 'export PATH="$HOME/bin:$PATH"' >> ~/.bashrc - run: name: Build application Docker image command: | docker build -f ./Dockerfile -t chezou/my-image:latest . - run: name: Test treasure-boxes workflows command: | set -x digdag run test_custom.dig

Building a Docker image on CircleCI, you can use it form digdag run command with the following workflow and Dockerfile.

# test_custom.dig+task: py>: test.show docker: image: "chezou/my-image:latest"

# DockerfileFROM python:3.7-slim-buster

RUN pip install tabula-py

CMD ["python3"]

Conclusion

Using CircleCI’s machine executor enables to use digdag run with digdag Docker executor.
It empowers us to do run through test for new Docker image with existing workflow on CircleCI

You can try it with this GitHub repo:

The first conference of Operational Machine Learning: OpML ‘19

Mon, 03 Jun 2019 21:50:07 -0700

I attended OpML ’19 is a conference for “Operational Machine Learning” held at Santa Clara on May 20th.

The scope of this conference is varied and seems not to be specified yet, even if I attended it. I’ll borrow the description from the OpML website.

The 2019 USENIX Conference on Operational Machine Learning (OpML ’19) provides a forum for both researchers and industry practitioners to develop and bring impactful research advances and cutting edge solutions to the pervasive challenges of ML production lifecycle management. ML production lifecycle is a necessity for wide-scale adoption and deployment of machine learning and deep learning across industries and for businesses to benefit from the core ML algorithms and research advances.

Overview of the conference

The number of attendees was 210, they came from LinkedIn, Microsoft, Google, Airbnb, Facebook, etc.
The target of “Operational Machine Learning” is diverse. I thought it focuses on MLOps things such as reproducibility, ML DSL for productionization, visualization, stakeholder management, but there are many talks about ML for system, system utilization optimization, SRE for ML, hardware accelerator, etc.
There is a contrast between tech giants, e.g. Google, Uber, Facebook, Airbnb, Microsoft, and LinkedIn, and other followers. While ML lead companies are talking about their OSSs or ML infrastructures, following companies tend to talk about their specific use case or their solutions (those speakers seems to be small ML ventures).

Some interesting talks

Keynote: Ray: A Distributed Framework for Emerging AI Applications

Current target of Machine Learning is pattern recognition, but Jordan said decision-making will be the future of ML/AI
Creating a “recommendation market” is the key

MLOp Lifecycle Scheme for Vision-based Inspection Process in Manufacturing

A challenge for defeat recognition by an image in edge applied for Samsung smartphone.
They need to inference for 3000 GB images/day.
The team structure which involves product inspectors and product managers is interesting

From [https://www.usenix.org/sites/default/files/conference/protected-files/opml19\_slides\_lim.pdf](https://www.usenix.org/sites/default/files/conference/protected-files/opml19_slides_lim.pdf)

From

AIOps: Challenges and Experiences in Azure

Anomaly detection and diagnosis with lambda architecture for Azure
Disk failure prediction for Azure which introduces proactively live to migrate the workloads to a healthy disk

How the Experts Do It: Production ML at Scale

A panel discussion for ML infrastructures

Lead and moderator: Joel Young, LinkedIn

Panelists:

Sandhya Ramu, Director, AI SRE, LinkedIn
Andrew Hoh, Product Manager, ML Infra and Applied ML, AirBNB
Aditya Kalro, Engineering Manager, AI Infra Services and Platform, Facebook
Faisal Siddiqi, Engineering Manager, Personalization Infrastructure, Netflix
Pranav Khaitan, Engineering Manager, Personalization and Dialog ML Infra, Google

The important thing to keep top level is

the lead time from experiment to production
Flows build for production with involving different team
Not everything is the highest priority. Metrics, dashboards are important

Cost of run/train vs Agility

It’s hard to find down streaming use cases. (Airbnb)
Monitor model resource usage
Keep ML infrastructure extremely flexible
Hard to force using a single framework

What are the important things for your ML platform?

Facebook

Reliability
Scalability
Developer productivity

Agility (Available libraries etc)
Enabling the latest technology
Cost and impact of Machine Learning

Netflix

How quickly/many A/B test we can do
How rapid new researcher can do?

Airbnb

Business impact
# of users for the infrastructures
How many inferences/scoring is done?
Availability, scalability, cost, and long-term decision making

Google

Innovation aspect
How can the ML infrastructure system will empower the next 5 yrs products?

Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform

TFX provides a library for recording and retrieving metadata for ML: ML Metadata

From [https://www.usenix.org/system/files/opml19papers-baylor.pdf](https://www.usenix.org/system/files/opml19papers-baylor.pdf)

From

Disdat: Bundle Data Management for Machine Learning Pipelines

Talk about OSS for ML pipeline and data versioning.

Predictive Caching@Scale

Traffic prediction for CDN (Akamai)
Interesting cache strategy with covering prediction error

Ruby for Data Science and Machine Learning

Tue, 23 Apr 2019 20:10:28 -0700

I attended held at Fukuoka from Apr 18 to Apr 21. This year’s RubyKaigi was a really great opportunity for me to know the possibility of Data Science and Machine Learning for Ruby.

Data Science and Ruby

As many of you may know, Ruby is widely known for web application with such as Ruby on Rails, but there is another momentum of Ruby or non-Python language. Here is the list of the sessions about Data Science.

A Deep Learning Adventure \[[repo](https://github.com/nusco/deep_learning_adventure)\](talked by Paolo Perrotta, the author of Metaprogramming Ruby!)
Reducing ActiveRecord memory consumption using Apache Arrow

Center of data science with Ruby

There is three core software supporting these movements:

Apache Arrow
Numo/Cumo
Red Chainer (Deep Learning framework ported from Chainer, implemented in Python)

Apache Arrow is a cross-language data structure for in-memory data. Kohei Sutou, the creator of Red Arrow, Ruby binding of Apache Arrow, who is a Japanese PMC of Apache Arrow. He has also been organizing an initiative called Red Data tools, monthly developer meet-ups for Ruby data tools. The meetup drives Ruby data ecosystem, especially for beginners. I heard from mrkn, a Ruby committer, that Arrow is trying to implement data manipulations those pandas does as C++ code. That means, calculations of tabula style data, a.k.a. DataFrame can be done in Apache Arrow’s Table format so that Ruby would be able to be suitable for data manipulation.

Another essential thing is Numo, which enables to handle numeric array like Numpy and is the fundamental part of DS/ML execution. Cumo is the GPU version of Numo and 75 times faster than Numo for the hello world for Deep Learning, a.k.a. MNIST. The talk about Cumo suggested that many Deep Learning related executions depend on CUDA so that scripting languages can be just a wrapper of them.

Red Chainer enables Deep Learning tasks, but it seems still young. Rather than that, can be a great tool, which allows to inference/predict with pre-trained models with PyTorch, Chainer, or any other frameworks which can export ONNX, the intermediate format of DL.

So, how will be the Ruby data science going on?

Looking at those momenta of Apache Arrow and Cumo, I feel the data science on Ruby would become much easier since the core problems which are related to execution speed can be hidden into C++/GPU layer. And using Menoh-Ruby can be a good opportunity for Ruby on Rails applications to serve prediction results on Ruby!

Red Data tools also create opportunities for many software engineers to jump into ML/DS world. One of my friends told me why he started working on Red Data tools that he wanted to change his field, and it’s an excellent area to join.

If you interested in this movement, let’s join !

A recent update of tabula-py

Sun, 17 Feb 2019 08:26:00 -0800

Photo by [Joshua Rawson-Harris](https://unsplash.com/@joshrh19?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)

Photo by on

This article is published last December. I’m planning to bump up the next version of tabula-py within few weeks.

Note

(Note: Oct 7th, 2019) As of Oct. 2019, I launched and for tabula-py. The FAQ would be good place to execute accurate extraction.

This is my first post on patreon. Apologies for delayed announcement of recent update of tabula-py. I will introduce the key features of updates.

Use Tabula app template

Tabula app has feature to reuse same bounding box for extraction. tabula-py now load and extract with tabula app’s template.

dfs = tabula.read_pdf_with_template(
 './examples/data.pdf',
 './examples/data.tabula-template.json',
 pandas_options={'header': 0})

Support file-like object

Like many python libraries, tabula-py has been able to .

# With file-like object 
pdf\_path = ‘tests/resources/data.pdf’
with open(pdf\_path, ‘rb’) as f:
 df = tabula.read_pdf(f)

# With pathlib 
from pathlib import Path
pdf_path = 'tests/resources/data.pdf'
df = tabula.read_pdf(Path(pdf_path))

Allow multiple area option

As of tabula-java v1.0.2, tabula can handle multiple area option.

pdf_path = 'tests/resources/MultiColumn.pdf'
# Relative area 
df_relative = tabula.read_pdf(
 pdf_path, pages=1,
 area=[[0, 0, 100, 50], [0, 50, 100, 100]], relative_area=True)

# Absolute area 
 df_absolute = tabula.read_pdf(
 pdf_path, pages=1, area=[[0, 0, 451, 212], [0, 212, 451, 425]])

Tip: Get table position

This is not a new feature, but I think it might be helpful for some PDFs.
Detailed post:

read_pdf with JSON contains position info, so you can get the table position as follows:

In [5]: tables = read_pdf("./examples/data.pdf", output_format="json", page=2)
In [9]: top = tables[0]['top']
In [10]: left = tables[0]['left']
In [11]: bottom = tables[0]['height'] + top
In [12]: right = tables[0]['width'] + left
In [13]: top, bottom, left, right
Out[13]: (0.0, 528.8800048828125, 0.0, 564.8800048828125)

If you have any question, ask on !

Use Markdown document on brand new PyPI

Mon, 16 Apr 2018 21:21:33 -0700

Yesterday, PyPI was renewed to the next-generation site. It is modern and stylish one.

told me that , which was accepted Feb. 2018, allows us for a document on PyPI to use not only reStructuredText but also other formats such as Markdown.

So I enabled my Markdown document on brand-new PyPI.

Upgrade Python packages (if necessary)

We can use Markdown with setuptools . Let’s upgrade you python packages if needed. Without that, Markdown description will not be rendered appropriately.

$ python -m pip install --upgrade pip $ pip install –upgrade wheel
$ pip --version pip 10.0.0 from c:\\users\\chezo\\documents\\source\\tabula-py\\venv\\lib\\site-packages\\pip (python 3.6) $ pip list
Package Version Location
-—————- ———– ————————————–
(…snip…)
setuptools 38.1.0
(…snip…)
wheel 0.31.0

Modify setup.py

If you’ve already used README.md as a long description on PyPI, all you have to do is to add long_description_content_type to setup.py as follows:

long_description=open('README.md').read(),
long_description_content_type="text/markdown",

You can see the full description of the PR :

Build a wheel and upload with twine

Now, you can build a wheel and upload with twine.

$ python setup.py bdist_wheel
$ twine upload dist/*

The Markdown document was rendered!

CAVEAT: I didn’t upgrade PyPI because it is too much to bump up for just rendering Markdown. I .

References

Python basics: package management

Tue, 29 Aug 2017 19:31:15 -0700

Python is a very famous programming language for machine learning. In this article, I will introduce basic Python environment.

Glossary

I will introduce basic terms about Python package management.

pip: A tool for package installation. It retrieves Python packages from . pip is gem command of Ruby.
virtualenv: Package isolation tool for Python. It has similar function with bundler of Ruby, but it also has the function to change Python versions over 2.x and 3.x.
venv: It is an official tool for package isolation introduced from Python 3.3. But, if you want to use Python 2.x or you are Debian/Ubuntu user, I recommend you to use virtualenv.

venv switches with a command like python3.5 -m venv some-awesome-env, so it can’t handle over Python 2 and 3. venv installed by Debian/Ubuntu installs useless dependencies for other OSs, so I’m an Ubuntu user so I don’t use venv.

These are common tool sets for many Pythonistas. They are of , a working group that maintains many of the relevant projects in Python packaging.

There is one more tool that is for the specific purpose.

conda: conda is a tool for package management for scientific computation developed by , Inc. It can manage not only Python but also R. PyData community loves conda.

I use , but I recommend you to know the pros/cons of conda and virtualenv/venv and chose write tool for your purpose.

Installation of Python

Since it is 2017, Python beginners should use the latest version of Python 3. However, there are some cases to use Python 2.x for some painful reasons.

If you need to install Python 2 and 3, you can install multiple Python with package management tools like apt or yum. In Ubuntu, you can install Python 2.7 with apt install python-dev, and you can install Python 3.6 via apt install python3-dev.

After installation, you can see the Pythons under /usr/bin:

/usr/bin/python #<- 2.7
/python2 #<- 2.7
/python2.7 #<- 2.7
/python3 #<- 3.6
/python3.6 #<- 3.6

If you’re macOS user, you can install both Python 2 and 3 via brew install or port install.

For Windows users, you can install Python 2 and 3 using official installer or Chocolatey. From Python 3.6 for Windows, there is py command that switches Python version.

Caution: Never try to keep using System Python. System Python is often old, and it depends on system critical system such as yum. If you run sudo pip install carelessly, there is a risk of destroying the environment of the OS itself.

Package management

As I mentioned, you should not do sudo pip install awesome-package. Hence, Many important systems depend on system Python, don’t use sudo pip.

If you’re a venv user, this tutorial will help you.

For virtualenv users, I will write a tutorial of virtualenv. It is a translation of the document written by aodag.

Why should we use virtualenv/venv?

virtualenv avoids:
- Conflicting Python packages with system Python
- Conflicting packages between projects
- Losing sight of which project depends on those packages

Install virtualenv

First, you can install virtualenv under user home directory.

$ wget https://bootstrap.pypa.io/get-pip.py
$ export PATH=”~/.local/bin/:$PATH”
$ python get-pip.py --user
$ pip install virtualenv --user
\# Windows user can isntall just via \`pip install\`
\> pip install virtualenv

With --user option, you can install packages under user directory.

virtualenv can create a Python virtual environment. Creating the environment under the project root is common.

Run virtualenv as follows:

$ virtualenv venv -p python3.6

then, you can get virtual environment.

Since Python packages will be installed under the venv directory, don’t forget to add venv directory into .gitignore.

$ source venv/bin/activate
(venv) $
\# For Windows
\> . venv/Script/activate

Install Python packages via pip

You can install packages via pip. After activating virtualenv/venv, pip will install packages under venv directory.

(venv) $ pip install pyramid

If you want to install the specific version of the package, you can set version number:

(venv) $ pip install pyramid==1.8.1

Without version number, pip will install latest stable version.

You can list installed packages with pip list command.

(venv) $ pip list
numpy (1.13.1)
pandas (0.20.3)
pip (9.0.1)
pkginfo (1.4.1)
pytest (3.2.0)
python-dateutil (2.6.1)
pytz (2017.2)
wheel (0.29.0)

Managing package version

From pip 7.1, we can fix version of packages with constraints.txt. Using pip freeze command, you can list packages with a version number.

(venv)$ pip freeze -l
numpy==1.13.1
pandas==0.20.3
pkginfo==1.4.1
pytest==3.2.0
python-dateutil==2.6.1
pytz==2017.2
(venv)$ pip freeze -l > constraints.txt

You should list your required packages into requirements.txt,

(venv)$ cat requirements.txt
pandas
numpy

Then you can install required packages as follows:

(venv)$ pip install -r requirements.txt -c constraints.txt

Levelaging wheelhouse

Modern Python package is distributed by wheel format, which is the binary type format. There is another format, sdist, which is the source type format and it requires compile from source if it depends on native codes. I highly recommend using wheel format, because it is faster installation than sdist without compilation and even if you have an offline environment which unable to connect PyPI you can deploy the project easily.

Put all dependent .whl format package files under wheelhouse directory, you can install as follows:

$ pip install -r requirements.txt -c constraints.txt -f wheelhouse — no-index

-w or --wheel-dir option allows you to set wheel directory. -f or--find-links option uses wheelhouse directory primary.--no-index option prevent to connect PyPI.

If you want to export all the dependencies into wheelhouse directory, you can use pip wheel command.

$ pip wheel -r requirements.txt -c constraints.txt -w wheelhouse

Should I use conda?

Anaconda is a Python distribution for scientific computing such as machine learning. Anaconda suit consists of Anaconda, which includes the recommended package and Miniconda, which is the minimum environment for conda and you can install only necessary packages yourself. Anaconda sometimes includes heavy packages. It used to include Django, so check the default package and use it properly.

Unlike virtualenv, Anaconda can create its original virtual environment. Characteristically, using the --copy option makes it possible to copy system level libraries, .so, etc. without creating symbolic links. If you archive a set of virtual environments with zip or tar, you can use it on other machines.

$ conda create -n myenv --copy python=3.6
$ conda activate myenv

In other words, libraries, which are managed by OS level package management tools such as apt, are also managed by conda. Conda has its own package repository different from PyPI and upload binaries for each OS on it. Since the same package, such as OpenCV, is registered in the repository by multiple users, you should care which package is the best one.

In many machine learning books, it is often written that conda can be used, but I think that it is better not to use it much outside Windows.

The reasons are as follows:

In 2017, wheel is de facto for the binary package format, so conda’s original purpose, handling scientific packages like numpy, or Scipy, can be done without conda.
conda will replace commands such as openssl/curl/python in macOS / Linux System (strictly speaking, conda will pass PATH first) \[[issue](https://github.com/ContinuumIO/anaconda-issues/issues/1119)\]
Package developers are often not conda users, and they seem to be asked for support in an environment that they do not normally use, such as JRuby or Rubyinius (or Windows specific trouble).
In the conda world, it is difficult to pass information that should be included in a build of a native extension (such as Cython dependence)

So I recommend using conda for Windows users or people do not develop heavily but want to experience machine learning. Or, put Miniconda under pyenv control. I use conda under Docker environment.

However, we can not install the package like Scipy on Windows via pip install, you need to download wheel on your own. I think that this point is better for honest conda.

Historical details are detailed in . In short, because old binary format egg was not good, conda was created.

Conclusion

I introduced installation of Python and how to manage Python packages. I think we can manage Python packages via virtualenv/venv well without conda, but there is good case for conda to pack some environment with system libraries.

References

Original Japanese document:

Why OSS based machine learning is good?

Wed, 02 Aug 2017 20:56:59 -0700

This article is translation of .

After releasing of TensorFlow, the movement of OSS-based machine learning is accelerating. , the creator of Keras, says the essential point of this change. I think his phrase is enough, but in this article, I would like to organize why open source machine learning is great, and what recent trends are.

tl;dr

Machine learning and deep learning frameworks have become standard things for software engineers
Since arXiv becomes very famous, many papers are published before peer review of international conferences. This change made easier for other companies to validate the algorithm.
Many researchers have been started to study machine learning, machine learning researches in academia become Red Oceanic.
The strategy, “Make a great algorithm, but the implementation is secret” becomes a thing of the past.

Halcyon days

Five or ten years ago, almost all players working on advanced machine learning were in laboratories such as universities or large enterprises, or some advanced companies. In particular, the amount of data with a label was smaller than the present, and many researchers had been improving the performance by researching algorithms, by feature engineering.

Many researchers from academia studied state-of-the-art machine learning, posted to international conferences. Most of the insights were shared after peer review. Implementation was not shared as much as now, and each researcher had to reimplement the preceding research from scratch. A typical cycle for releasing new algorithms was a half year, in some cases more than a year.

There were few open source machine learning libraries/frameworks like Weka. scikit-learn, , was not famous among software engineers. Many of us used libraries with single/few algorithms such as libsvm and liblinear.

Fast moving era

As of 2017, people who work in machine learning have significantly increased compared with 10 years ago. The center of machine learning has been moved from academia to companies with large data. In particular, software engineers, who have never worked on machine learning, entering deep learning world. I was surprised to hear that my friend of the community who had never worked on machine learning in business had started working on Deep Learning. The reasons for this movement are 1) it became general for companies to store large data that can be used for machine learning,
2) excellent machine learning frameworks have been increased, and 3) the GPU power leverage Deep Learning for efficient calculation.

Many open source libraries became popular not only in the frameworks of Deep Learning such as TensorFlow, Chainer, MXNet, Caffe 2, PyTorch but also by XGBoost, Lightgbm, which are famouse among kaggler. scikit-learn is also common tool as a framework to experiment with multiple algorithms.

The rise of “open papers”

This movement is supported by machine learning competition site “kaggle”, and by a place to post open papers called “arXiv”. (There is discussion arxiv does not have a peer review process and quality is not assured. So can we call the document as a research paper? But, in this post, I will call the research paper style report as “paper”)

The following article describes the number of paper submissions related to machine learning (especially Deep Learning) submitted to arXiv. According to this article, it is pointed out that the number of papers related to machine learning has more than quadrupled in 2017 compared to five years ago.

Papers of arXiv are posted every day. It means, state-of-the-art results from such as Google, Facebook, Microsoft, etc. are published more and more before peer review. This is a challenge for the central laboratories of the traditional large enterprises to research and develop cutting edge algorithms of machine learning itself. Those companies usually set targets for a year or half a year. There is also criticism of “just adding parts”, but it is clear that the speed of developing machine learning algorithms is significantly fast.

In the field of machine translation, the breakthrough in deep learning was encoder-decoder and attention. The subsequent papers are not interesting, “I just put existing parts here.” I can’t understand why these papers come to the top conference.

Recently, for those who read new arXiv’s paper day and night, there is an system called “ariXiv Times” to better check new arrival documents.

Open papers accelerates Open source machine learning

This March, a paper about “Deep Forest” was published at arXiv, and it became a hot topic with the author claims that “performance is better than Deep Learning”.

This method proposed in this paper, about one week (2017/3/5) after the publication (2017/2 / 28), R implementation came up and Python implementation came out after R one. A discussion was made with the following LightGBM issue on GitHub, and it came out that there was not reproducibility of the article, they can’t confirm the performance.

It is a symbolic event where the OSS implementation of the paper
appeared within a week after published in arXiv and the discussion of the community began.

I hear that it is increasing that the number of international conferences that require disclosing the implementation when a paper is submitted.

Conclusion

It is an essential task to develop the machine learning algorithm. Thanks to open papers, ML competition web site, and fast implementation of new algorithms as an OSS, we can adopt state-of-the-art knowledge into the business rapidly.

IMHO, it is becoming fun to focus on where we can make use of ML in business rather than developing the algorithm itself.

In other words, now, it is too hard to say “special machine learning algorithms that only our company can do”. Of course, people in academia will push these cutting-edge initiatives if they can prepare data. What is the evidence that one company invents a better algorithm quickly than most state-of-the-art people from tech giants like Google, Facebook, Microsoft, etc.? That is the reason for the strength of open source based machine learning.

Among academia, there is a famous phrase, “ ”, it means that we should thank previous research then we can go on to the next step. Even in machine learning based on open source, we can not ignore this phrase. We cannot ignore giants.

How to run Cloudera Director on your macOS/Windows 10

Tue, 01 Aug 2017 20:12:31 -0700

Cloudera Director is a provisioning tool for CDH and Cloudera Enterprise. We can launch cluster with Web GUI or CLI tool. Using Cloudera Director CLI tool, you can manage your cluster with configuration file, that enables you to manage configurations with git. In this article, I will introduce how to install Cloudera Director into your local macOS or Windows 10.

For usage of Cloudera Director, see also the document.

Install Cloudera Director on you macOS with homebrew

If you’re homebrew user, you can install Cloudera Director easily.

$ brew tap chezou/cloudera $ brew install cloudera-director-server

Then, you can launch/terminate Cloudera Director as follows:

# Start Cloudera Director Server background
$ cloudera-director-server-start
# After launching director server, you can open with http://locahost:7189/

# Stop Cloudera Director Server background
$ cloudera-director-server-stop

Install Cloudera Director on you Windows 10

If you are Windows 10 user, you can install Ubuntu as the .

Launch bash on windows, then run as follows:

Make sure to get not IP address of Windows but Ubuntu’s one.

Use Docker image

If you don’t want to install your machine directly, you can use Docker image of Cloudera Director.

After installation of Docker, run following commands then your Director will launch.

$ git clone https://github.com/tsuyo/cloudera-boot$ cd cloudera-boot$ . bin/cloudera-boot.sh # load several functions/aliases$ cb-build # may take a while# set you secrets

You can launch a Director server or use client as well. To get further information, see also README.md.

Simple way to distribute your private Python packages within your organization

Sun, 23 Jul 2017 09:21:40 -0700

[https://www.irasutoya.com/2017/05/blog-post\_22.html](https://www.irasutoya.com/2017/05/blog-post_22.html)

This article is a translation of , originally written by in Japanese. I translated it with his permission. This article is aimed to know simple ways to prepare internal Python package host like a .

Methods

Include your packages in your git repository
Publish a directory including your packages via HTTP server
Build a local PyPI-equivalent server

It is a high-cost way to create a local PyPI-equivalent server (translator note: like ), and I don’t think there is no need to do so, I will describe first two options.

Include your packages in your Git repository

If your packages are required for a particular project, it is straightforward to contain them in the Git repository. You can put them in the directory named wheelhouse, which comes from the name of the previous default directory created by pip wheel. (translator note: this method is assumed you to know wheel. If not, and would be helpful.)If you put the private package foo in the wheelhouse, you can install as follows:

$ pip install foo -f wheelhouse

Note that -f is the short option for --find-links, with that option, pip will search packages in the directory first, then fall back to pypi.

Publish a directory including your packages via HTTP server

We can use--find-link option to search not only local directory but also a remote server via http. If you have a package used by multiple projects, this method will help you.

The easiest way to distribute your packages with this method is executing python -m http.server with Python 3.x (or python -m SimpleHTTPServer with Python 2.7) on the wheelhouse directory. This simple server provides directory listings so that we can just use--find-links to use the directory. Make sure to open http://localhost:8000 that you can see the list of files under the wheelhouse directory via a web browser.

To install foo package via HTTP server you launched, you can execute as follows:

$ pip install foo -f http://localhost:8000

Since this is a simple server, for production, it is good to put them in cloud storage such as AWS S3, you should check the way for directory listings, or you can use Apache with DirectoryIndex enabled.

Conclusion

I recommend these methods because they are simple and no need to prepare the dedicated application server.

tabula-py now able to extract remote PDF and multiple tables at once

Sat, 27 May 2017 19:18:39 -0700

Note

(Note: Oct 7th, 2019) As of Oct. 2019, I launched and for tabula-py. The FAQ would be good place to execute accurate extraction.

tabula-py is a Python library which enables you to extract tables from PDF into pandas DataFrames. Today, I released v0.8.0. In this post, I will introduce improvements after previous post of tabula-py. If you don’t familiar with tabula-py, you can see previous one.

Change Notes

Able to read remote PDF passing URL
\[Experimental\]Add multiple_tables mode
Add batch conversion method:convert_into_by_batch()
Add encoding option
Add java_options
Will deprecate read_pdf_table() method

I will explain important features.

Read remote PDF passing URL

If you want extract a DataFrame from the internet, you can extract remote PDF without downloading it manually.

read_pdf("https://github.com/tabulapdf/tabula-java/raw/master/src/test/resources/technology/tabula/12s0324.pdf")

\[Experimental\] Add “`multiple_tables"` mode

tabula-py is a simple wrapper of tabula-java, it was hard to handle multiple tables in a page. But now, you can extract multiple tables in a page using multiple_tables option.

read_pdf('tests/resources/data.pdf', pages=2, multiple_tables=True)

This function create a list of DataFrames via JSON from tabula-java, so if tabula-java’s JSON format will change, the output could be broken. If you see CParserError , try to set multiple_tables option.

Add batch conversion method: “`convert_into_by_batch()"`

After tabula-java v0.9.2, we can extract tables from PDF by batch. You can use this function through convert_into_by_batch() method.

convert_into_by_batch(path_to_dir, output_format='csv')

You should set directory path of PDFs, not the specific pdf path.

tabula-py extracts tables same directory as input files.

TODOs

There are several problems those may be fixed after releasing of tabula-java 0.9.3. e.g) Handling embedded font, including Japanese…

Waiting for your collaboration!

If you have any troubles with tabula-py, please file . I don’t want to receive emails because the answer will not share to other people. Make sure fill , it will reduce many costs for me to solve the problem.

An easy way to get URL list of your Medium publication

Mon, 01 May 2017 19:01:01 -0700

I imported blog posts from own Wordpress but I have to redirect old articles to Medium manually. There is Wordpress plugin which enables you to redirect articles, but it requires URL mapping in CSV format. When you want to get Medium publication’s URL list, you may use , but officially, it . We need , but I couldn’t get the post list. In this article, I will show you how to get URL list of your Medium publication easily.

Note: I tried this method with under 150 articles publication. It might not work with huge number of articles.

How-to

You can use following Python script, after showing whole articles with accesing /latest of a publication. For example, after opening , you can get whole contents with scrolling down and down and down…

sparkavro: Manupilate Apache Avro file with sparklyr

Sun, 26 Mar 2017 05:02:01 -0700

I created a simple extension to handle Apache Avro file. It is just a simple wrapper of DataBrick’s . It is listed in .

Installation

Use {devtools} to install sparkavro.

devtools::install_github(“chezou/avrospark”)

Simple usage

You can read and write Avro file as follows:

library(sparklyr)
library(sparkavro)
sc <- spark_connect(master = “spark://HOST:PORT”)
df <- spark_read_avro(sc, “test_table”, “/user/foo/test.avro”)
spark_write_avro(df, “/tmp/output”)

This is the very first version, so there might be bugs especially around options. If you find any bug, please raise on the .

How to connect secure Impala cluster from RStudio on macOS with implyr

Sat, 25 Mar 2017 14:35:45 -0700

Impala is very fast SQL-on-Hadoop, and it will enhance your R experience with , a based interface for created by . I will show you how to setup connection to Kerberized Impala cluster with implyr from local macOS. You can find my GitHub repo as follows:

Setting up ODBC environment for macOS

Install unixODBC with homebrew

First, we will install to handle Impala with ODBC. In R world, ODBC is preferred to connect Impala because of its performance and compatibility. Let’s install unixODBC with homebrew.

$ brew install unixodbc

Download and install the latest version of the Impala ODBC driver from Cloudera

You can download .

Configure your .odbc.ini and .odbcinst.ini

After installing Impala ODBC driver for macOS, basic configuration templates can be found in /opt/cloudera/impalaodbc/Setup/.

cp /opt/cloudera/impalaodbc/Setup/odbc.ini ~/.odbc.ini
cp /opt/cloudera/impalaodbc/Setup/odbcinst.ini ~/.odbcinst.ini

Before using following setting, you must replace HOST and KrbRealm with appropriate ones. Let’s modify your .odbc.ini as follows:

\[ODBC\]

# Specify any global ODBC configuration here such as ODBC tracing.

\[ODBC Data Sources\]

Impala=Cloudera ODBC Driver for Impala

\[Impala\]

# Description: DSN Description.
# This key is not necessary and is only to give a description of the data source.
Description=Cloudera Impala ODBC Driver DSN

# Driver: The location where the ODBC driver is installed to.
Driver=/opt/cloudera/impalaodbc/lib/universal/libclouderaimpalaodbc.dylib

# The DriverUnicodeEncoding setting is only used for SimbaDM
# When set to 1, SimbaDM runs in UTF-16 mode.
# When set to 2, SimbaDM runs in UTF-8 mode.
#DriverUnicodeEncoding=2

# Values for HOST, PORT, KrbFQDN, and KrbServiceName should be set here.
# They can also be specified on the connection string.
HOST=

\[REPLACE\_YOUR\_IMPALA\_HOST\]

PORT=21050
Schema=default

# The authentication mechanism.
# 0 — No authentication (NOSASL)
# 1 — Kerberos authentication (SASL)
# 2 — Username authentication (SASL)
# 3 — Username/password authentication (NOSASL or SASL depending on UseSASL configuration)
AuthMech=1

# Set to 1 to use SASL for authentication.
# Set to 0 to not use SASL.
# When using Kerberos authentication (SASL) or Username authentication (SASL) SASL is always used
# and this configuration is ignored. SASL is always not used for No authentication (NOSASL).
UseSASL=1

# Kerberos related settings.
KrbFQDN=_HOST
KrbRealm=

\[REPLACE\_YOUR\_REALM\]

KrbServiceName=impala

# Username/password authentication with SASL settings.
UID=
PWD=

# Set to 0 to disable SSL.
# Set to 1 to enable SSL.
SSL=1
CAIssuedCertNamesMismatch=1
TrustedCerts=/opt/cloudera/impalaodbc/lib/universal/cacerts.pem

# If you use SSL with AllowSelfSignedServerCert, you can set this configuration.
#AllowSelfSignedServerCert=1

# Specify the proxy user ID to use.
#DelegationUID=

# General settings
TSaslTransportBufSize=1000
RowsFetchedPerBlock=10000
SocketTimeout=0
StringColumnLength=32767
UseNativeQuery=0

After setting up the .odbc.ini , your application will refer this setting with appropriate DSN name, like Impala in this case.

Check the configuration

After configuration, you should kinit with your principal.

$ kinit $USER@YOUR_REALM

You should replace `$USER` and `YOUR_REALM` with the appropriate REALM.

Before using RStudio on you mac, you can check configuration with `isql` command.

\[tablename\]

|
| quit |
| |
+ — — — — — — — — — — — — — — — — — — — -+
SQL>

Implyr Example

After setting .odbc.ini you can connect secure Impala cluster with {implyr}. For instance, We will visualize .

First, install R packages.

install.packages(c(“implyr”, “odbc”, “DBI”, “dplyr”, “ggplot2”, “ggExtra”))

Then, connect the Impala cluster.

library(implyr)
library(odbc)
drv <- odbc::odbc()
impala <- src_impala(
drv = drv,
dsn = “Impala”
)

If your .odbc.ini is configured properly, you can connect to Impala cluster.

Let’s visualize the airports data. In this case, we assume the data is in u_ariga database, so that we will change database using SQL use u_ariga.

library(DBI)
# Change database
dbExecute(impala, “use u_ariga”)
dbGetQuery(impala, “show tables”)
airports <- tbl(impala, “airports_pq”)

# Show the head of airports data
View(airports)

airports %>% filter(latitude < 35) %>% count()
#903

Finally, we will show a joint histogram of longitude and latitude.

airports_by_geo <- airports %>% select(longitude, latitude) %>% collect()

library(ggplot2)

p <- ggplot(airports_by_geo, aes(longitude, latitude)) + geom_point() + theme_classic()
ggExtra::ggMarginal(p, type = “histogram”)

Conclusion

{implyr} is a great package for Impala and dplyr but it is pretty young project. If you find some problems, why don’t you post into ?

Visualize your massive data with Impala and Redash

Fri, 10 Feb 2017 21:14:44 -0800

is a famous OSS visualization tool, which enables to visualize your data with SQL. It supports , fast SQL-on-Hadoop suitable for BI tools and exploratory analysis. With Impala, you can .

In this post, we connect to Impala from Redash and visualize data.

Set up Redash

You can set up Redash with various way. This time, I use . Then, you can access with your browser with admin/admin.

Add Data Source of Impala

After clicking Database icon, you can add data sources.

This time, I set configurations as follows:

Example configuration

Type: Impala
Database: default
Host: hostname of Impala daemon
Ldap_password/user: (empty)
Port: 21050 (default port)
Please specify beeswax or hiveserver2: hiveserver2
Timeout: 3600
Use_ldap: (empty)

Now, you can select Impala as a data source.

Result of Impala query

tabula-py: Extract table from PDF into Python DataFrame

Sun, 08 Jan 2017 21:09:08 -0800

Note

(Oct 7th, 2019) As of Oct. 2019, I launched a and for tabula-py. The FAQ would be good place to execute accurate extraction.

Screenshots in this article is based on the old version interface. See the latest version example in the Colab notebook.

Today, I released tabula-py 0.3.0, which extracts table from PDF into Python pandas’s DataFrame.

It is simple wrapper of and it enables you to extract table into DataFrame or JSON with Python. You also can extract tables from PDF into CSV, TSV or JSON file.

is a tool to extract tables from PDFs. It is GUI based software, but tabula-java is a tool based on CUI. Though there were , , and bindings of tabula-java, before tabula-py there isn’t any Python binding of it. I believe PyData is a great ecosystem for data analysis and that’s why I created tabula-py. If you are familiar with R, I highly recommend to use , which has the most richest bindings including rich GUI.

You can install tabula-py via pip:

pip install tabula-py

With tabula-py, you can get DataFrame with read_pdf() method.

example of read_pdf()

example of read_pdf()

You can also extract tables as JSON format:

example of JSON

You can extract tables into a file like JSON, CSV or TSV with convert_into() method.

You can see more examples in Jupyter notebook.

I hope you will enjoy data wrangling with tabula-py. Any feedback would be welcome!

Waiting for your collaboration!

If you have any trouble with tabula-py, please file . I don’t want to receive emails because the answer will not share with other people. Make sure to fill , it will reduce many costs for me to solve the problem. Or, I also check StackOverflow. You can ask about it.

Livy & Jupyter Notebook & Sparkmagic = Powerful & Easy Notebook for Data Scientist

Thu, 29 Dec 2016 22:15:23 -0800

livy is a REST server of Spark. You can see , . is one of the most popular notebook OSS within data scientists. Using sparkmagic + Jupyter notebook, data scientists can execute ad-hoc Spark job easily.

Why livy is good?

According to , livy has features like:

Have long running SparkContexts that can be used for multiple Spark jobs, by multiple clients
Share cached RDDs or Dataframes across multiple jobs and clients
Multiple SparkContexts can be managed simultaneously, and they run on the cluster (YARN/Mesos) instead of the Livy Server for good fault tolerance and concurrency
Jobs can be submitted as precompiled jars, snippets of code, or via Java/Scala client API
Ensure security via secure authenticated communication
Apache License, 100% open source

Why livy + sparkmagic?

is a client of livy using with Jupyter notebook. When we write Spark code at our local Jupyter client, then sparkmagic runs the Spark job through livy. Using sparkmagic + Jupyter notebook, data scientists can use Spark from their own Jupyter notebook, which is running on their localhost. We don’t need any Spark configuration getting from the CDH cluster. So we can execute Spark job in a cluster like running on a local machine.

diagram from

Requirements

Spark Cluster

Cloudera Director is nice to prepare
Install git and maven
I tried CDH 5.7 with CentOS 7

2. Local jupyter client

virtualenv and virtualenvwrapper is awesome

Preparation

In order to use livy with sparkmagic, we should install livy into the Spark gateway server and sparkmagic into local machine.

Install R

$ sudo yum install -y epel-release$ sudo yum install -y R

Build livy

$ git clone git@github.com:cloudera/livy.git$ cd livy$ mvn -Dspark.version=1.6.0 -DskipTests clean package

Because of failing test at that time, I added -DskipTests to build.

Run livy

Set environment variables as follows:

$ export SPARK_HOME=/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/spark$ export HADOOP_CONF_DIR=/etc/hadoop/conf

Add the following configuration into livy.conf:

livy.server.session.factory = yarn

Let’s run livy server

$ ./bin/livy-server

Open another terminal and check the server

$ curl localhost:8998/sessions{"from":0,"total":0,"sessions":[]}

As livy’s Default port number is 8998, we should open or forward the port.

Prepare sparkmagic in local machine

Install sparkmagic by following the :

$ pip install sparkmagic$ jupyter nbextension enable --py --sys-prefix widgetsnbextension

Then install wapper kernel. Do pip show sparkmagic and you can see the Location info. In the following example, Location is /Users/ariga/.virtualenvs/ibis/lib/python3.5/site-packages.

$ pip show sparkmagic---Metadata-Version: 2.0Name: sparkmagicVersion: 0.2.3Summary: SparkMagic: Spark execution via LivyHome-page: https://github.com/jupyter-incubator/sparkmagic/sparkmagicAuthor: Jupyter Development TeamAuthor-email: jupyter@googlegroups.orgInstaller: pipLicense: BSD 3-clauseLocation: /Users/ariga/.virtualenvs/ibis/lib/python3.5/site-packagesRequires: ipywidgets, pandas, ipython, requests, mock, autovizwidget, numpy, nose, ipykernel, notebook, hdijupyterutilsClassifiers: Development Status :: 4 - Beta Environment :: Console Intended Audience :: Science/Research License :: OSI Approved :: BSD License Natural Language :: English Programming Language :: Python :: 2.6 Programming Language :: Python :: 2.7 Programming Language :: Python :: 3.3 Programming Language :: Python :: 3.4$ cd /Users/ariga/.virtualenvs/ibis/lib/python3.5/site-packages$ jupyter-kernelspec install sparkmagic/kernels/sparkkernel$ jupyter-kernelspec install sparkmagic/kernels/pysparkkernel

Copy the into ~/.sparkmagic/config.json and modify it.

Run jupyter notebook

Before running jupyter, I recommend checking the connection from the local machine to the livy server.

$ curl YOUR_HOSTNAME:8998/sessions

Launch jupyter notebook and create PySpark notebook (of course you can use Spark)

$ jupyter notebook

The example notebook is here

In the nbviewer, we can not see the result of SQL, but we can visualize the result of SQL with %%sql magic command. That’s awesome :)

If you use %%local, you can use local Python libraries such as scikit-learn, seaborn etc, with received results from PySpark.

References

Text-to-speech based on deep learning for Web site using Amazon Polly and Ruby

Wed, 30 Nov 2016 22:00:02 -0800

Amazon Polly, Text-to-speech service from AWS was announced at today ‘s re:Invent. Amazon Polly is speech synthesize system based on deep learning.

\[updated\] I added generated speech of this article.

\[updated2\] I created simple CLI tools and rubygems of polly

The great thing about Amazon Polly is that we can use TTS easily with AWS CLI. The price is free for up to 5 million characters a month, if over that limitation, it is very cheap with $ 0.000004/character. If you synthesize [Adventures of Huckleberry Finn](https://en.wikipedia.org/wiki/Adventures_of_Huckleberry_Finn), it costs about only $2.4.

Here is the example code of Polly with AWS CLI tool.

$ aws polly synthesize-speech \
–output-format mp3 –voice-id Joanna \
–text “Hello my name is Joanna.” \
joanna.mp3

As of December 1, 2016, they support the following 24 languages mainly in European languages.

Icelandic
Italian
Welsh
Dutch
Swedish
Spanish (Castile)
Spanish (USA)
Danish
Turkish
German
Norwegian
French
French (Canada)
Portuguese
Portuguese (Brazil)
Polish
Romanian
Russian
Japanese
English (India)
English (Welsh)
English (Australia)
English (US)
English (UK)

I think Japanese speech sounds very natural. Sometime it will be a strange accent, but if I register a word with Lexicon, we can improve the quality by myself. Japanese sample voice as following:

I often find interesting articles in Medium, but since reading long English article is a bit tough for non native English speaker like me. So I came up with if I made the article to voice, I would listen it easily. That’s why I wrote the code to convert articles to speech with Ruby like following:

There are some important restrictions of API:

The number of characters per API is 1500 characters
Long voice is truncated after 5 minutes

Building predictive Model with Ibis, Impala and scikit-learn

Fri, 14 Oct 2016 14:10:31 -0700

tl;dr

visualizing 20M data (famous movie rating data) with
build predictive model for movie favor with scikit-learn
/

What is Ibis?

Ibis is a bridge between Python and Big Data. Ibis enables pandas handling Big Data.

architecture of Ibis

For more detail, see Wes’s presentation.

As you know, pandas is known as a killer application for data analysis. In my previous job, which is known as a developer of , many Rails developer attracted with pandas and Jupyter notebook for sharing analysis result.

Why Ibis?

pandas loads data on memory, so we have to filter with some SQL before analyzing. But we actually want to get insight and handle without SQL.

Preparation

Impala cluster

CDH 5.7 with Cloudera Director 2.1
table is created with parquet on S3

required port

impalad node’s 21050 port
NN’s 50070 port

Ibis

Python 3.5
using wheel and virtualenv, I didn’t use anaconda

Notebook

Full notebook repo is . I also executed same code for Redshift, but several dialects prevent execution…

FAQ

What is the difference between PySpark?

Easy to setup. It is just like connecting DB
Fast x10. So that we can x10 experiences. It makes us innovations!
We can rapid prototyping with Ibis.

Which is prefer to build model Ibis + scikit-learn or Spark + MLlib?

It depends on data size.
. Netflix uses R in order to model filtered data such as specific country, and they use Spark for global model.

Blog Posts | Democratizing Data

TikTok kept letting strangers create accounts with my email, so I filed PIPEDA and CASL complaints

TL;DR

What is PIPEDA and CASL?

What happened

March 2026: contacting TikTok directly

April 2026: the same problem, again

Two regulatory routes in Canada

CASL → CRTC

PIPEDA → OPC

The form’s automatic stop

Recovering through the Information Centre

Filing the formal complaint

Existing OPC × TikTok context

What I did, in summary

A note on AI-assisted research

Embedding workflow templates in skills: shifting the LLM's role from "generation" to "rendering"

The dream of LLM-powered ML workflow generation

The rise of Claude Code and agent-friendly CLIs

What the agent-friendly CLI made possible

Turning ML workflow templates into skills

Jinja2 templates for digdag workflows

OpenAPI as the contract with the agent

Skill-creator agent vs. skill-user agent

Wrapping up

Migrated from Pages CMS to Sveltia CMS

Motivation

What I did

Impressions

Between Principal and Glue Work

Introduction

The 4 Staff Archetypes

1. Tech Lead

2. Architect

3. Solver

4. Right Hand

Breaking Down My Work by Archetypes

Why Does the Scope of Staff+ Expand?

Reflecting on My Career, or Redefining “Glue Work”

To Those With the Same Struggles

Tackling Review Fatigue by Document Driven Agentic Coding

Proxy Prompting Through Other Humans

Lack of Self-Review, or the “Eyes Glazing Over” Problem

Big Bang Commit

The Difference in Incentive Structures Between the Generator and the Reviewer

How to Approach Code Reviews Mostly Generated by LLM

1. The Generator Reviews Their Own Code as Someone Else’s on GitHub

2. Document-Driven Development

How to Approach Document Reviews Mostly Generated by LLM

Continue Struggling

Configured Pages CMS

Machine Learning Project and Scrum

Disclaimer

Research Method

Anticipated Challenges

Using Technical Stories

Business Value in ML Projects

Differences in Task-Level Development Flow

How to Implement Scrum in ML Projects

1. Stop Using Scrum and Use Kanban

2. Use Hypothesis Based Stories (Modifying Scrum)

Aside: What Does a Product Owner Do in ML Projects?

Conclusion

Migrated From Netlify to Cloudflare Pages

Scrape Notion and convert into PDF

Recipe on Notion is good, if it’s printable

Scrape Notion with Python

tabula-py 2.8.0 now uses jpype to launch JVM

How fast is it?

Caveats

Challenges for releasing v2.8.0

The test issue with different java_options

Read the docs default behavior change

4 Steps to Release a CLI in Python

Create a project by using poetry

Create a CLI with Click/Cloup

Use poetry-dynamic-versioning for version management

Introduce GitHub Actions to release the package to PyPI

Create data lineage from Trino/Hive queries in digdag log with Python

What’s data lineage?

The test issue with different `java_options`