168 lines
9.2 KiB
HTML
168 lines
9.2 KiB
HTML
<!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<meta charset="utf-8" />
|
|
<meta http-equiv="X-UA-Compatible" content="IE=edge"><title>Pickle 5 Madness with MLFlow and Python 3.6/3.7 - Brainsteam</title><meta name="viewport" content="width=device-width, initial-scale=1">
|
|
<meta itemprop="name" content="Pickle 5 Madness with MLFlow and Python 3.6/3.7">
|
|
<meta itemprop="description" content="Solving 'unsupported pickle protocol: 5' when trying to load mlflow models"><meta itemprop="datePublished" content="2021-01-14T11:42:28+00:00" />
|
|
<meta itemprop="dateModified" content="2021-01-14T11:42:28+00:00" />
|
|
<meta itemprop="wordCount" content="416"><meta itemprop="image" content="https://brainsteam.co.uk/2021/01/14/pickle-5-madness-with-mlflow/images/feature.jpg">
|
|
<meta itemprop="keywords" content="machine-learning,python,ai,devops,mlops," /><meta property="og:title" content="Pickle 5 Madness with MLFlow and Python 3.6/3.7" />
|
|
<meta property="og:description" content="Solving 'unsupported pickle protocol: 5' when trying to load mlflow models" />
|
|
<meta property="og:type" content="article" />
|
|
<meta property="og:url" content="https://brainsteam.co.uk/2021/01/14/pickle-5-madness-with-mlflow/" /><meta property="og:image" content="https://brainsteam.co.uk/2021/01/14/pickle-5-madness-with-mlflow/images/feature.jpg"/><meta property="article:section" content="posts" />
|
|
<meta property="article:published_time" content="2021-01-14T11:42:28+00:00" />
|
|
<meta property="article:modified_time" content="2021-01-14T11:42:28+00:00" />
|
|
|
|
<meta name="twitter:card" content="summary_large_image"/>
|
|
<meta name="twitter:image" content="https://brainsteam.co.uk/2021/01/14/pickle-5-madness-with-mlflow/images/feature.jpg"/>
|
|
<meta name="twitter:title" content="Pickle 5 Madness with MLFlow and Python 3.6/3.7"/>
|
|
<meta name="twitter:description" content="Solving 'unsupported pickle protocol: 5' when trying to load mlflow models"/>
|
|
<link href='https://fonts.googleapis.com/css?family=Playfair+Display:700' rel='stylesheet' type='text/css'>
|
|
<link rel="stylesheet" type="text/css" media="screen" href="https://brainsteam.co.uk/css/normalize.css" />
|
|
<link rel="stylesheet" type="text/css" media="screen" href="https://brainsteam.co.uk/css/main.css" />
|
|
|
|
<link id="dark-scheme" rel="stylesheet" type="text/css" href="https://brainsteam.co.uk/css/dark.css" />
|
|
|
|
<script src="https://brainsteam.co.uk/js/feather.min.js"></script>
|
|
|
|
<script src="https://brainsteam.co.uk/js/main.js"></script>
|
|
</head>
|
|
|
|
<body>
|
|
<div class="container wrapper">
|
|
<div class="header">
|
|
|
|
<div class="avatar">
|
|
<a href="https://brainsteam.co.uk/">
|
|
<img src="/images/avatar.png" alt="Brainsteam" />
|
|
</a>
|
|
</div>
|
|
|
|
<h1 class="site-title"><a href="https://brainsteam.co.uk/">Brainsteam</a></h1>
|
|
<div class="site-description"><p>The irregular mental expulsions of a PhD student and CTO of Filament, my views are my own and do not represent my employers in any way.</p><nav class="nav social">
|
|
<ul class="flat"><li><a href="https://twitter.com/jamesravey/" title="Twitter" rel="me"><i data-feather="twitter"></i></a></li><li><a href="https://github.com/ravenscroftj" title="Github" rel="me"><i data-feather="github"></i></a></li><li><a href="/index.xml" title="RSS" rel="me"><i data-feather="rss"></i></a></li></ul>
|
|
</nav></div>
|
|
|
|
<nav class="nav">
|
|
<ul class="flat">
|
|
|
|
<li>
|
|
<a href="/">Home</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="/tags">Tags</a>
|
|
</li>
|
|
|
|
<li>
|
|
<a href="https://jamesravey.me">About Me</a>
|
|
</li>
|
|
|
|
</ul>
|
|
</nav>
|
|
</div>
|
|
|
|
<div class="post">
|
|
<div class="post-header">
|
|
|
|
<div class="meta">
|
|
<div class="date">
|
|
<span class="day">14</span>
|
|
<span class="rest">Jan 2021</span>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="matter">
|
|
<h1 class="title">Pickle 5 Madness with MLFlow and Python 3.6/3.7</h1>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="markdown">
|
|
<figure>
|
|
<img src="images/feature.jpg"
|
|
alt="A jar of pickles by Ksenia Charnaya"/> <figcaption>
|
|
<p>A jar of pickles by <a href='https://www.pexels.com/photo/crop-unrecognizable-person-with-jar-of-pickled-zucchini-3952045/'>Ksenia Charnaya</a></p>
|
|
</figcaption>
|
|
</figure>
|
|
|
|
<p>I recently came across an infuriating problem where an <a href="https://www.mlflow.org/docs/latest/python_api/mlflow.pyfunc.html">MLFlow python model</a> I had trained on one system using Python <code>3.6</code> would not load on another system with an identical version of Python.</p>
|
|
<p>The exact problem was that when I ran <code>mlflow models serve -m <url/to/model/in/bucket></code> the service would crash saying that the model could not be unserialized because <code>ValueError: unsupported pickle protocol: 5</code>.</p>
|
|
<p>A quick bit of searching shows that this error happens when something is pickled in Python 3.8 which uses pickle protocol 5 by default and loaded by a system running an earlier version of Python 3 (3.6 or 3.7) which only support pickle protocol up to v4.</p>
|
|
<p>Under the covers mlflow uses <a href="https://github.com/cloudpipe/cloudpickle">cloudpickle</a>, a library that provides extended pickle support including the ability to pickle lambda functions and functions/classes defined interactively in the <code>__main__</code> module of your program or in a Jupyter notebook. By default <code>cloudpickle</code> uses the highest version of pickle protocol available in your python implementation (by checking <a href="https://docs.python.org/3/library/pickle.html#pickle.HIGHEST_PROTOCOL">pickle.HIGHEST_PROTOCOL</a> constant) - this makes sense for most use cases where you want to serialize objects and pass them around within the same Python setup - as a rule of thumb, more recent protocols are better performing/more efficient.</p>
|
|
<p>However this is a mystery because I’m running Python <code>3.6.12</code> on both systems which does not support protocol 5, so how is it that cloudpickle is using this version to write the models? I still haven’t worked this out and if anyone knows please get in touch because it is driving me mad!</p>
|
|
<p>Luckily for us, although the use of v5 is puzzling, there is a solution. The <a href="https://pypi.org/project/pickle5/">pickle5</a> library provides version 5 support that is backwards compatible with Python 3.6 and 3.7. Furthermore, <a href="https://github.com/dask/distributed/pull/3849">cloudpickle will automatically detect and load this library if it is available</a>. Therefore all we need to do is install <code>pickle5</code> in our MLFLow serving environment to make this issue go away.</p>
|
|
<p>The easiest way to make sure pickle5 is available to your server is by adding it to your conda env when you save your model to MLFlow:</p>
|
|
<div class="highlight"><pre style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">
|
|
model = SomeScikitLearnModel()
|
|
model.fit(X,y)
|
|
|
|
conda_env = mlflow.pyfunc.get_default_conda_env()
|
|
conda_env[<span style="color:#a31515">'dependencies'</span>].append({<span style="color:#a31515">'pip'</span>: [
|
|
<span style="color:#a31515">'pickle5'</span>
|
|
<span style="color:#a31515">'scikit-learn==0.23.2'</span>
|
|
<span style="color:#008000">#... some other dependencies</span>
|
|
]})
|
|
|
|
mlflow.sklearn.log_model(model, <span style="color:#a31515">"model"</span>, conda_env=conda_env)
|
|
</code></pre></div><p>Note: I already checked and <code>pickle5</code> is not installed in the first environment but the Conda base version of Python on that system is <code>3.8.3</code> so I think there must be some weird leakage of the conda paths going on when I train my model.</p>
|
|
|
|
</div>
|
|
|
|
<div class="tags">
|
|
|
|
|
|
<ul class="flat">
|
|
|
|
<li><a href="/tags/machine-learning">machine-learning</a></li>
|
|
|
|
<li><a href="/tags/python">python</a></li>
|
|
|
|
<li><a href="/tags/ai">ai</a></li>
|
|
|
|
<li><a href="/tags/devops">devops</a></li>
|
|
|
|
<li><a href="/tags/mlops">mlops</a></li>
|
|
|
|
</ul>
|
|
|
|
|
|
</div><div id="disqus_thread"></div>
|
|
<script type="text/javascript">
|
|
(function () {
|
|
|
|
|
|
if (window.location.hostname == "localhost")
|
|
return;
|
|
|
|
var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
|
|
var disqus_shortname = 'brainsteam';
|
|
dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
|
|
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
|
|
})();
|
|
</script>
|
|
<noscript>Please enable JavaScript to view the </a></noscript>
|
|
<a href="http://disqus.com/" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a>
|
|
</div>
|
|
</div>
|
|
<div class="footer wrapper">
|
|
<nav class="nav">
|
|
<div>2021 © James Ravenscroft 2020 | <a href="https://github.com/knadh/hugo-ink">Ink</a> theme on <a href="https://gohugo.io">Hugo</a></div>
|
|
</nav>
|
|
</div>
|
|
|
|
|
|
<script type="application/javascript">
|
|
var doNotTrack = false;
|
|
if (!doNotTrack) {
|
|
window.ga=window.ga||function(){(ga.q=ga.q||[]).push(arguments)};ga.l=+new Date;
|
|
ga('create', 'UA-186263385-1', 'auto');
|
|
|
|
ga('send', 'pageview');
|
|
}
|
|
</script>
|
|
<script async src='https://www.google-analytics.com/analytics.js'></script>
|
|
<script>feather.replace()</script>
|
|
</body>
|
|
</html>
|