sweetcocoa / pop2piano

huggingface.co
Total runs: 9.8K
24-hour runs: 0
7-day runs: 690
30-day runs: 41
Model's Last Updated: August 29 2023
automatic-speech-recognition

Introduction of pop2piano

Model Details of pop2piano

POP2PIANO

Pop2Piano, a Transformer network that generates piano covers given waveforms of pop music.

Model Details

Pop2Piano was proposed in the paper Pop2Piano : Pop Audio-based Piano Cover Generation by Jongho Choi and Kyogu Lee.

Piano covers of pop music are widely enjoyed, but generating them from music is not a trivial task. It requires great expertise with playing piano as well as knowing different characteristics and melodies of a song. With Pop2Piano you can directly generate a cover from a song's audio waveform. It is the first model to directly generate a piano cover from pop audio without melody and chord extraction modules.

Pop2Piano is an encoder-decoder Transformer model based on T5 . The input audio is transformed to its waveform and passed to the encoder, which transforms it to a latent representation. The decoder uses these latent representations to generate token ids in an autoregressive way. Each token id corresponds to one of four different token types: time, velocity, note and 'special'. The token ids are then decoded to their equivalent MIDI file.

Model Sources

Usage

To use Pop2Piano, you will need to install the 🤗 Transformers library, as well as the following third party modules:

pip install git+https://github.com/huggingface/transformers.git
pip install pretty-midi==0.2.9 essentia==2.1b6.dev1034 librosa scipy

Please note that you may need to restart your runtime after installation.

Pop music to Piano
Code Example
  • Using your own Audio
>>> import librosa
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor

>>> audio, sr = librosa.load("<your_audio_file_here>", sr=44100)  # feel free to change the sr to a suitable value.
>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> processor = Pop2PianoProcessor.from_pretrained("sweetcocoa/pop2piano")

>>> inputs = processor(audio=audio, sampling_rate=sr, return_tensors="pt")
>>> model_output = model.generate(input_features=inputs["input_features"], composer="composer1")
>>> tokenizer_output = processor.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"][0]
>>> tokenizer_output.write("./Outputs/midi_output.mid")
  • Audio from Hugging Face Hub
>>> from datasets import load_dataset
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor

>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> processor = Pop2PianoProcessor.from_pretrained("sweetcocoa/pop2piano")
>>> ds = load_dataset("sweetcocoa/pop2piano_ci", split="test")

>>> inputs = processor(
...     audio=ds["audio"][0]["array"], sampling_rate=ds["audio"][0]["sampling_rate"], return_tensors="pt"
... )
>>> model_output = model.generate(input_features=inputs["input_features"], composer="composer1")
>>> tokenizer_output = processor.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"][0]
>>> tokenizer_output.write("./Outputs/midi_output.mid")
Example

Here we present an example of generated MIDI.

  • Actual Pop Music
  • Generated MIDI
Tips
  1. Pop2Piano is an Encoder-Decoder based model like T5.
  2. Pop2Piano can be used to generate midi-audio files for a given audio sequence.
  3. Choosing different composers in Pop2PianoForConditionalGeneration.generate() can lead to variety of different results.
  4. Setting the sampling rate to 44.1 kHz when loading the audio file can give good performance.
  5. Though Pop2Piano was mainly trained on Korean Pop music, it also does pretty well on other Western Pop or Hip Hop songs.

Citation

BibTeX:

@misc{choi2023pop2piano,
      title={Pop2Piano : Pop Audio-based Piano Cover Generation}, 
      author={Jongho Choi and Kyogu Lee},
      year={2023},
      eprint={2211.00895},
      archivePrefix={arXiv},
      primaryClass={cs.SD}
}

Runs of sweetcocoa pop2piano on huggingface.co

9.8K
Total runs
0
24-hour runs
298
3-day runs
690
7-day runs
41
30-day runs

More Information About pop2piano huggingface.co Model

pop2piano huggingface.co

pop2piano huggingface.co is an AI model on huggingface.co that provides pop2piano's model effect (), which can be used instantly with this sweetcocoa pop2piano model. huggingface.co supports a free trial of the pop2piano model, and also provides paid use of the pop2piano. Support call pop2piano model through api, including Node.js, Python, http.

sweetcocoa pop2piano online free

pop2piano huggingface.co is an online trial and call api platform, which integrates pop2piano's modeling effects, including api services, and provides a free online trial of pop2piano, you can try pop2piano online for free by clicking the link below.

sweetcocoa pop2piano online free url in huggingface.co:

https://huggingface.co/sweetcocoa/pop2piano

pop2piano install

pop2piano is an open source model from GitHub that offers a free installation service, and any user can find pop2piano on GitHub to install. At the same time, huggingface.co provides the effect of pop2piano install, users can directly use pop2piano installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

pop2piano install url in huggingface.co:

https://huggingface.co/sweetcocoa/pop2piano

Url of pop2piano

Provider of pop2piano huggingface.co

sweetcocoa
ORGANIZATIONS